Oct 31 14:45:18.404878 ip-10-0-139-115 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 14:45:18.766208 ip-10-0-139-115 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:18.766208 ip-10-0-139-115 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 14:45:18.766208 ip-10-0-139-115 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:18.766208 ip-10-0-139-115 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771313    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771394    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771398    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771401    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771404    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771407    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.771400 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771410    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771412    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771415    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771418    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771421    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771423    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771426    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771429    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771432    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771434    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771437    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771440    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771442    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771445    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771448    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771450    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771453    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771455    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771459    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.771669 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771463    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771466    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771469    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771472    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771474    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771477    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771480    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771488    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771491    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771494    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771496    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771499    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771501    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771504    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771507    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771510    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771513    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771515    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771518    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771521    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.772117 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771524    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771526    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771529    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771531    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771534    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771536    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771539    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771541    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771543    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771547    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771549    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771552    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771554    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771557    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771560    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771562    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771565    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771584    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771588    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.772618 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771592    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771595    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771598    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771601    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771603    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771606    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771608    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771611    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771615    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771618    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771621    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771623    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771626    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771628    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771631    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771637    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771641    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771645    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771650    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.773095 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771654    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771656    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771659    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771661    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771664    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771667    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771669    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771672    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771675    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771678    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771681    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771684    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771686    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771689    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771691    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771693    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771696    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771698    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771701    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.773583 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771704    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771707    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.771710    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772549    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772558    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772562    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772584    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772591    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772596    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772600    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772603    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772606    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772608    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772612    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772615    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772617    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772620    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772623    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772625    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.774025 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772628    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772630    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772633    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772635    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772638    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772641    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772643    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772646    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772648    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772651    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772653    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772656    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772658    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772662    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772665    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772668    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772670    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772673    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772675    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772678    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.774508 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772681    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772684    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772687    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772690    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772692    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772694    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772697    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772700    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772703    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772706    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772708    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772711    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772713    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772716    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772719    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772722    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772724    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772729    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772732    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772735    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.775031 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772737    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772740    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772742    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772745    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772747    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772749    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772753    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772755    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772758    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772760    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772762    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772765    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772767    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772770    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772772    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772775    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772777    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772780    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772782    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.775510 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772785    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772788    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772791    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772793    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772796    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772799    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772801    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772804    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772806    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772809    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772812    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772817    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772819    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772822    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772824    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772826    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772829    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772831    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772837    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.775986 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772839    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772842    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772844    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772846    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772849    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772852    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772854    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772857    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772859    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772862    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772864    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.772866    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773371    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773376    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773380    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773383    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773386    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773388    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773391    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773394    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.776454 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773396    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773399    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773404    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773407    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773409    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773412    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773414    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773417    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773419    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773422    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773425    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773429    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773431    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773434    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773436    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773439    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773441    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773444    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773446    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773449    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.776951 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773451    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773454    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773456    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773459    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773461    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773464    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773466    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773469    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773472    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773474    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773477    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773480    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773484    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773486    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773491    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773494    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773496    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773498    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773501    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.777433 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773504    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773506    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773509    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773511    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773515    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773519    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773522    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773525    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773529    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773532    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773534    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773537    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773539    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773542    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773544    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773547    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773549    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773552    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773555    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773557    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.777905 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773559    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773562    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773565    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773584    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773589    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773593    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773597    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773602    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773605    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773608    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773610    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773612    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773615    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773618    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773620    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773623    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773626    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773628    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773631    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773633    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.778368 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773636    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773638    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773641    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773643    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773645    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773648    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773650    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773652    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773655    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773658    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773660    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773663    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773665    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773668    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773671    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773674    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773676    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773679    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.773681    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774673    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 14:45:18.778869 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774684    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774695    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774700    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774704    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774708    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774712    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774716    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774719    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774723    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774727    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774730    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774733    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774736    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774739    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774742    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774745    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774748    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774754    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774757    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774760    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774763    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774766    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774770    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 14:45:18.779344 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774773    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774777    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774780    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774783    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774787    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774790    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774794    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774797    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774801    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774804    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774807    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774810    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774813    2572 flags.go:64] FLAG: --enable-server="true"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774816    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774821    2572 flags.go:64] FLAG: --event-burst="100"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774824    2572 flags.go:64] FLAG: --event-qps="50"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774827    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774830    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774833    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774837    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774840    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774843    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774846    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774849    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774852    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 14:45:18.779937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774855    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774858    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774861    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774864    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774866    2572 flags.go:64] FLAG: --feature-gates=""
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774870    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774874    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774877    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774880    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774883    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774886    2572 flags.go:64] FLAG: --help="false"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774889    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774892    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774895    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774898    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774901    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774905    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774908    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774911    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774913    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774916    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774919    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774922    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774925    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 31 14:45:18.780559 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774928    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774931    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774934    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774936    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774939    2572 flags.go:64] FLAG: --lock-file=""
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774942    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774945    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774947    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774953    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774956    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774959    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774961    2572 flags.go:64] FLAG: --logging-format="text"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774964    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774967    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774970    2572 flags.go:64] FLAG: --manifest-url=""
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774973    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774978    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774981    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774985    2572 flags.go:64] FLAG: --max-pods="110"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774987    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774991    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774994    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.774999    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775002    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775005    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 14:45:18.781197 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775008    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775015    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775019    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775022    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775025    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775028    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775031    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775035    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775038    2572 flags.go:64] FLAG: --port="10250"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775041    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775045    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-06ec943c9bfebf1d1"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775048    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775050    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775053    2572 flags.go:64] FLAG: --register-node="true"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775056    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775060    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775063    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775066    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775069    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775072    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775075    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775078    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775081    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775084    2572 flags.go:64] FLAG: --runonce="false"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775087    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775090    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775093    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 14:45:18.781926 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775096    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775099    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775103    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775106    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775109    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775112    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775115    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775118    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775120    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775123    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775126    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775129    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775135    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775138    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775141    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775145    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775147    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775150    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775153    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775156    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775158    2572 flags.go:64] FLAG: --v="2"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775163    2572 flags.go:64] FLAG: --version="false"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775167    2572 flags.go:64] FLAG: --vmodule=""
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775171    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775174    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 14:45:18.782624 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775313    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775317    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775320    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775322    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775325    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775328    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775331    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775333    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775336    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775339    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775342    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775345    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775348    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775350    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775353    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775355    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775358    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775361    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775363    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775367    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.783225 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775370    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775373    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775375    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775378    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775380    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775383    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775385    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775388    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775390    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775393    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775395    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775398    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775400    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775403    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775405    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775407    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775410    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775412    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775415    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775417    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.783720 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775420    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775423    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775426    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775429    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775431    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775434    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775436    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775439    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775441    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775444    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775446    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775450    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775453    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775455    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775458    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775460    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775462    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775465    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775467    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775470    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.784208 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775472    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775475    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775477    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775480    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775482    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775484    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775487    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775489    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775492    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775494    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775498    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775502    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775505    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775509    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775512    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775514    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775520    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775523    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775526    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775528    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.784733 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775531    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775533    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775536    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775539    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775542    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775544    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775547    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775551    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775554    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775557    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775560    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775562    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775565    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775582    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775584    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775596    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775599    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775601    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775604    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.785203 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775607    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775609    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775612    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775615    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775617    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775620    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775624    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.775651    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775690    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775695    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775705    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775709    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775712    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775714    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775717    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.785712 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775719    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775722    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775727    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775730    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775733    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775736    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775738    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775741    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775743    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775746    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775748    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775751    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775753    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775756    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775758    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775761    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775763    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775766    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775768    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775771    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.786067 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775773    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775777    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775780    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775783    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775788    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775790    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775793    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775795    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775798    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775801    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775804    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775806    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775809    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775811    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775815    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775818    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775821    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775823    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775827    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.786531 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775830    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775833    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775836    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775839    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775842    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775844    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775847    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775849    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775852    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775854    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775857    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775859    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775862    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775864    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775867    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775870    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775873    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775876    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775879    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.786994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775881    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775884    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775886    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775889    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775892    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775895    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775897    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775899    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775903    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775905    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775908    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775910    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775913    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775915    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775918    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775920    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775923    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775925    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775928    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775930    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.787447 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775932    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775935    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775937    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775940    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775942    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775945    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775947    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775950    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775953    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775955    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775959    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775962    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775964    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775966    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775969    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775972    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775978    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775981    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775983    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775986    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.787962 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.775989    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.782805    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.782822    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.783334    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.783341    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783398    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783403    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783406    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783409    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783412    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783416    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783420    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783427    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783431    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783434    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783436    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783439    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783442    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783445    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783447    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.788533 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783450    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783452    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783455    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783458    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783461    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783463    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783466    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783468    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783471    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783473    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783476    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783478    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783480    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783483    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783487    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783490    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783492    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783495    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783498    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783500    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.789061 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783503    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783505    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783507    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783510    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783512    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783515    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783517    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783520    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783522    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783525    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783527    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783530    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783533    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783535    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783538    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783541    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783543    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783545    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783548    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.789665 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783551    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783554    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783556    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783559    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783561    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783564    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783586    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783589    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783593    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783595    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783598    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783601    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783603    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783606    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783608    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783610    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783613    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783615    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783618    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.790476 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783621    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783624    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783626    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783629    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783631    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783633    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783636    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783638    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783641    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783643    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783647    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783651    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783653    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783656    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783660    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783664    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783667    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783670    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783672    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.791231 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783675    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783678    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783681    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783684    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783686    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783689    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783691    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783694    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783697    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783699    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783702    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783704    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783707    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783709    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.783739    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783892    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.792070 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783897    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783901    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783904    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783906    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783909    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783912    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783914    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783917    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783919    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783922    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783924    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783927    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783929    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783932    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783934    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783937    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783940    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783942    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783945    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.792518 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783948    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783950    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783953    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783956    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783958    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783961    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783963    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783965    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783968    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783970    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783973    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783975    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783977    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783980    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783982    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783984    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783987    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783989    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783992    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783994    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.793278 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783996    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.783998    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784001    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784003    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784005    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784008    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784010    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784013    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784015    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784018    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784020    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784023    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784027    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784030    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784034    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784037    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784039    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784042    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784044    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.793878 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784047    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784049    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784051    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784054    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784056    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784058    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784061    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784063    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784066    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784068    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784070    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784073    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784075    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784077    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784080    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784083    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784085    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784087    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784090    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.794337 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784092    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784094    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784097    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784099    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784102    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784105    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784107    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784110    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784113    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784117    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784121    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784123    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784126    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784128    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784131    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784133    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784136    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784138    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784140    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.794860 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784143    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784145    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784147    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784150    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784152    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784154    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784157    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784159    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784162    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.784184    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784214    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784218    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784221    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784224    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784226    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784229    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:18.795335 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784231    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784234    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784236    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784239    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784241    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784243    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784246    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784249    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784251    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784254    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784262    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784267    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784270    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784272    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784274    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784277    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784279    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784282    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784284    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784286    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:18.795794 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784289    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784292    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784294    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784298    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784301    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784304    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784307    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784310    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784313    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784315    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784318    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784323    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784325    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784328    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784330    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784332    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784335    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784337    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784340    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:18.796334 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784342    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784345    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784348    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784350    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784353    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784355    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784357    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784360    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784362    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784365    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784367    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784370    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784372    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784374    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784377    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784379    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784382    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784384    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784388    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:18.796937 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784392    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784395    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784398    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784401    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784404    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784408    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784410    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784412    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784415    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784417    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784420    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784422    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784425    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784427    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784429    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784432    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784434    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784437    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784439    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784442    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:18.797399 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784444    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784447    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784449    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784452    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784454    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784456    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784459    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784461    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784463    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784466    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784468    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784472    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784475    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784477    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784480    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784482    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784484    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784488    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784490    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:18.797902 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784493    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784495    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:18.784498    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.784552    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.787316    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.788082    2572 server.go:1014] "Starting client certificate rotation"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.788182    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.788228    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.789940    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 14:45:18.798389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.792416    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 14:45:18.802637 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.802617    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 31 14:45:18.805555 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.805540    2572 log.go:25] "Validated CRI v1 image API"
Oct 31 14:45:18.806365 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.806342    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 14:45:18.808743 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.808723    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a4b8cee-0da1-4113-ae4a-384be7747e44:/dev/nvme0n1p4 88e7f930-5261-495b-9f42-565a6aeb4a84:/dev/nvme0n1p3]
Oct 31 14:45:18.808795 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.808744    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 14:45:18.814497 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.814346    2572 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:18.813189871 +0000 UTC m=+0.303229309 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099775 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246b599922c9270a2278bfee833525 SystemUUID:ec246b59-9922-c927-0a22-78bfee833525 BootID:6ac9ca99-8f81-44c4-9da4-585c73cb9b64 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:71:ac:00:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:71:ac:00:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:19:bb:7f:83:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 14:45:18.814497 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.814492    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 14:45:18.814626 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.814601    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 14:45:18.815739 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815712    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 14:45:18.815882 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815741    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-115.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 14:45:18.815924 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815885    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 14:45:18.815924 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815892    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 14:45:18.815978 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815930    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 14:45:18.815978 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.815951    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 14:45:18.816660 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.816649    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 14:45:18.820319 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.820306    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 14:45:18.820453 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.820444    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 14:45:18.821001 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.820991    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 14:45:18.821039 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.821007    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 14:45:18.821039 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.821020    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 14:45:18.821039 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.821030    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 31 14:45:18.821039 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.821038    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 14:45:18.821406 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.821395    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 14:45:18.823452 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.823435    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 14:45:18.827272 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.827259    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 14:45:18.828027 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828014    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 14:45:18.828062 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828046    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 14:45:18.828201 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828192    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 14:45:18.828235 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828209    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 14:45:18.828235 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828215    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 14:45:18.828235 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828221    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 14:45:18.828235 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828227    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 14:45:18.828235 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828232    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 14:45:18.828726 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828717    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 14:45:18.828755 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828728    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 14:45:18.828755 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828735    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 14:45:18.828755 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828741    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 14:45:18.828755 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828755    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 14:45:18.828878 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.828764    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 14:45:18.829339 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.829331    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 14:45:18.829369 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.829340    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 14:45:18.833174 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.833161    2572 server.go:1263] "Started kubelet"
Oct 31 14:45:18.833259 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.833231    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 14:45:18.833304 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.833229    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 14:45:18.836388 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.836365    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 14:45:18.836893 ip-10-0-139-115 systemd[1]: Started Kubernetes Kubelet.
Oct 31 14:45:18.839172 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.839149    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 14:45:18.840529 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.840509    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 14:45:18.841294 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.841276    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 14:45:18.843516 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.843495    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 14:45:18.844468 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.844451    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 14:45:18.844605 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.844563    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found"
Oct 31 14:45:18.845094 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.844979    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 14:45:18.845094 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.844996    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 14:45:18.845094 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.844978    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 14:45:18.845289 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845114    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 14:45:18.845289 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845123    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 14:45:18.845461 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845447    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 14:45:18.845499 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845462    2572 factory.go:55] Registering systemd factory
Oct 31 14:45:18.845499 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845469    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 31 14:45:18.845699 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845687    2572 factory.go:153] Registering CRI-O factory
Oct 31 14:45:18.845764 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845702    2572 factory.go:223] Registration of the crio container factory successfully
Oct 31 14:45:18.845764 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845721    2572 factory.go:103] Registering Raw factory
Oct 31 14:45:18.845764 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.845736    2572 manager.go:1199] Started watching for new ooms in manager
Oct 31 14:45:18.846479 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.846463    2572 manager.go:319] Starting recovery of all containers
Oct 31 14:45:18.855607 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.855588    2572 manager.go:324] Recovery completed
Oct 31 14:45:18.859618 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.859536    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:18.861327 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861309    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:18.861415 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861339    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:18.861415 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861348    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:18.861892 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861879    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 31 14:45:18.861949 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861892    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 14:45:18.861949 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861919    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:18.862051 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.861922    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 14:45:18.862543 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.862529    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:18.862635 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.862561    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:18.862635 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.862593    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:18.864212 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.864200    2572 policy_none.go:50] "Start"
Oct 31 14:45:18.864272 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.864216    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 14:45:18.864272 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.864226    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 14:45:18.866375 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.866365    2572 policy_none.go:44] "Start"
Oct 31 14:45:18.905185 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.905140    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 14:45:18.907726 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.907712    2572 manager.go:341] "Starting Device Plugin manager"
Oct 31 14:45:18.907798 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.907782    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 14:45:18.907798 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.907795    2572 server.go:87] "Starting device plugin registration server"
Oct 31 14:45:18.908077 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.908065    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 14:45:18.908115 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.908080    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 14:45:18.908201 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.908183    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 14:45:18.908583 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.908302    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 14:45:18.908583 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.908314    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 14:45:18.908784 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.908761    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 14:45:18.908829 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.908815    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-115.us-west-1.compute.internal\" not found"
Oct 31 14:45:18.975607 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.975560    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 14:45:18.977261 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.977241    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 14:45:18.977261 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.977258    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 14:45:18.977603 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.977279    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 14:45:18.977603 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.977310    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 14:45:18.977603 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.977379    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-115.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 14:45:18.977603 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:18.977389    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-115.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 14:45:18.978272 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:18.977360    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-115.us-west-1.compute.internal.18e3a4799ba67ed1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-115.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:18.832967377 +0000 UTC m=+0.323006811,LastTimestamp:2026-10-31 14:45:18.832967377 +0000 UTC m=+0.323006811,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-115.us-west-1.compute.internal,}"
Oct 31 14:45:19.009161 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.009015    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:19.009995 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.009979    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:19.010073 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.010010    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:19.010073 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.010021    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:19.010073 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.010049    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.059138 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.059067    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 14:45:19.059138 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.059111    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ddsjg"
Oct 31 14:45:19.078338 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.078311    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"]
Oct 31 14:45:19.078407 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.078375    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:19.079194 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.079178    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:19.079261 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.079208    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:19.079261 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.079219    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:19.081492 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.081480    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:19.081673 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.081657    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.081720 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.081696    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:19.082215 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082201    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:19.082306 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082212    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:19.082306 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082235    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:19.082306 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082238    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:19.082306 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082246    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:19.082306 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.082254    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:19.084456 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.084437    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.084548 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.084469    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:19.085121 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.085105    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:19.085195 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.085139    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:19.085195 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.085153    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:19.113432 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:19.113410    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.115983 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:19.115966    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.132688 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.132666    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:19.133988 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.133974    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:19.146276 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.146251    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.146351 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.146282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.146351 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.146299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.232471 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.232445    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:19.233410 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.233395    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:19.233898 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.233882    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.244687 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.244664    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247242 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247315 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247315 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247324    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247324    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.247389 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.247346    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.305650 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.305625    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:45:19.305778 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.305724    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.363295 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.363228    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:45:19.416335 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.416302    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.418973 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.418956    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:19.788129 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.788108    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 14:45:19.788639 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.788307    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 14:45:19.788639 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.788321    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 14:45:19.788639 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.788336    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 14:45:19.788639 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:19.788308    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-115.us-west-1.compute.internal.18e3a4799d573150\": read tcp 10.0.139.115:60690->13.59.157.100:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-115.us-west-1.compute.internal.18e3a4799d573150  default   5259 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-115.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-115.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:18 +0000 UTC,LastTimestamp:2026-10-31 14:45:19.079195307 +0000 UTC m=+0.569234742,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-115.us-west-1.compute.internal,}"
Oct 31 14:45:19.821970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.821935    2572 apiserver.go:52] "Watching apiserver"
Oct 31 14:45:19.844444 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:19.844423    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 14:45:20.065100 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.065007    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:19 +0000 UTC" deadline="2028-05-10 19:05:54.260558585 +0000 UTC"
Oct 31 14:45:20.065100 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.065044    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13372h20m34.195518293s"
Oct 31 14:45:20.093947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.093928    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 14:45:20.094254 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.094236    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hmc4m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp","openshift-cluster-node-tuning-operator/tuned-s2smk","openshift-image-registry/node-ca-gjwxr","openshift-multus/multus-swww9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-lnk8v","kube-system/konnectivity-agent-ktm59","kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal","openshift-dns/node-resolver-bhqss","openshift-multus/network-metrics-daemon-64bfw","openshift-network-diagnostics/network-check-target-ljbk9","openshift-network-operator/iptables-alerter-ccjhc","openshift-ovn-kubernetes/ovnkube-node-x4sfm"]
Oct 31 14:45:20.096762 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.096746    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.098808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.098790    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.098903 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.098857    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:20.100879 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.100863    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.102827 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.102809    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.104933 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.104918    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.106995 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.106976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.107080 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.107038    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:20.113071 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.113051    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.116997 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.116982    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.118546 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.118531    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.120782 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.120766    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:20.120842 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.120829    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:20.124668 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.124647    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.126497 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.126482    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.145933 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.145915    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 14:45:20.148706 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.148685    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.148706 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.148687    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.148839 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.148802    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\""
Oct 31 14:45:20.148875 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.148862    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 14:45:20.148908 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.148882    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 14:45:20.151698 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.151685    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 14:45:20.151851 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.151840    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 14:45:20.152072 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.152060    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\""
Oct 31 14:45:20.153180 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.153231 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153231 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153204    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153231 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153219    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153326 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153239    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153326 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153326 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153417 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153417 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.153417 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.153517 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153433    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.153517 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153453    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.153517 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153517 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153517 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153540    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153559    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmb6c\" (UniqueName: \"kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153653    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.153727 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153756    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\""
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153753    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153840    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153841    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153882    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153896    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153911    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kcv\" (UniqueName: \"kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153961    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.153996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpnm\" (UniqueName: \"kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.154045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154046    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpkn\" (UniqueName: \"kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154097    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6tx\" (UniqueName: \"kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154111    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jwm\" (UniqueName: \"kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154185    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28t9l\" (UniqueName: \"kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154267    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154402    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154456    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.154808 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154480    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154592    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154623    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154652    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2c8\" (UniqueName: \"kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.155359 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.154996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155048    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155200    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zqg\" (UniqueName: \"kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155717    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155812    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\""
Oct 31 14:45:20.155970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.155838    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.246055 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.246029    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 14:45:20.246164 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.246067    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdqtp"
Oct 31 14:45:20.250322 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250293    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 14:45:20.250445 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250335    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\""
Oct 31 14:45:20.250552 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250536    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.250669 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250589    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\""
Oct 31 14:45:20.250775 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250761    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 14:45:20.250821 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250802    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\""
Oct 31 14:45:20.250864 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250821    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 14:45:20.250983 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.250970    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.251094 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251082    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 14:45:20.251177 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251166    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.251222 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251193    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 14:45:20.251268 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251244    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.251268 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251261    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 14:45:20.251356 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251328    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\""
Oct 31 14:45:20.251422 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251409    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.251482 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251468    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\""
Oct 31 14:45:20.251531 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251488    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 14:45:20.251593 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251562    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.251636 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251605    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 14:45:20.251695 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251682    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 14:45:20.251749 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.251718    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 14:45:20.255452 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255431    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kcv\" (UniqueName: \"kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255475    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255499    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpnm\" (UniqueName: \"kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.255557 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255546    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255561    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255586    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255642    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpkn\" (UniqueName: \"kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255689    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6tx\" (UniqueName: \"kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.255696    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.255784    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.755740132 +0000 UTC m=+2.245779583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255791    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255802    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jwm\" (UniqueName: \"kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.255871 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255866    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28t9l\" (UniqueName: \"kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255867    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255894    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255952    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.255987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256042    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256051    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256054    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256066    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256084    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256123    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.256495 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256192    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256206    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256233    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256302    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256302    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256317    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256330    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256353    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256358    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256405    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.257393 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256435    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256387    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256484    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256485    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256505    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256596    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2c8\" (UniqueName: \"kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256618    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256656    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256680    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256690    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256703    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.258266 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256819    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256915    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256943    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256948    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256968    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.256995    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257020    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zqg\" (UniqueName: \"kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257039    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257051    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257077    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257080    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257082    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259037 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257101    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257129    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257129    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257172    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257176    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257218    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257210    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257225    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257264    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257266    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257285    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257308    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.259740 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257360    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257382    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257495    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257517    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257536    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.257621    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.257666    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.757651029 +0000 UTC m=+2.247690466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257674    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257708    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257538    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257767    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257781    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmb6c\" (UniqueName: \"kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260219 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257847    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257882    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257878    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257913    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257932    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257925    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257968    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257976    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.257995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258040    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258148    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258206    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258229    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.260731 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258269    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258366    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.258844    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.259527    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.259850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.260008    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.260122    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.261329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.260592    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.311064 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.311034    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:20.311064 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.311065    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:20.311064 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.311079    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:20.311354 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.311147    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.811127157 +0000 UTC m=+2.301166581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:20.314130 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314100    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpnm\" (UniqueName: \"kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.314368 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314348    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zqg\" (UniqueName: \"kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.314446 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314426    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2c8\" (UniqueName: \"kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.314673 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314652    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28t9l\" (UniqueName: \"kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.314739 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314658    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6tx\" (UniqueName: \"kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.314739 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314689    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jwm\" (UniqueName: \"kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.314817 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314749    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpkn\" (UniqueName: \"kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.314817 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.314793    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kcv\" (UniqueName: \"kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.315981 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.315938    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmb6c\" (UniqueName: \"kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.420350 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.420317    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swww9"
Oct 31 14:45:20.448927 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.448899    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:45:20.465726 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.465698    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:45:20.481337 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.481313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:45:20.492954 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.492194    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:20.510290 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.510268    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:45:20.510596 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.510541    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57afef5546aa1919f861928dcb50ce6.slice/crio-45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515 WatchSource:0}: Error finding container 45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515: Status 404 returned error can't find the container with id 45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515
Oct 31 14:45:20.511037 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.511004    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381c174e_6f38_4afb_a8f1_08514439fa50.slice/crio-9ed339d3a6048e229999a4ca0cfe291475f00f1980343ae2bd9751f5ad0b32e2 WatchSource:0}: Error finding container 9ed339d3a6048e229999a4ca0cfe291475f00f1980343ae2bd9751f5ad0b32e2: Status 404 returned error can't find the container with id 9ed339d3a6048e229999a4ca0cfe291475f00f1980343ae2bd9751f5ad0b32e2
Oct 31 14:45:20.511783 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.511765    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebed5d09_55bf_4935_8ea9_55360332d683.slice/crio-e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b WatchSource:0}: Error finding container e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b: Status 404 returned error can't find the container with id e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b
Oct 31 14:45:20.512994 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.512978    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c519a9e746dec2e3a5ff69d67b2ad.slice/crio-085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb WatchSource:0}: Error finding container 085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb: Status 404 returned error can't find the container with id 085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb
Oct 31 14:45:20.518120 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.518106    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:45:20.519555 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.519535    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc686b7_618e_47ee_b28e_35f1cbee4212.slice/crio-7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3 WatchSource:0}: Error finding container 7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3: Status 404 returned error can't find the container with id 7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3
Oct 31 14:45:20.532073 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.532053    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:45:20.535189 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.535173    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:45:20.553249 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.553232    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:20.561790 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:20.561771    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63e0003_a810_483a_8ab0_4e01ea21f4e9.slice/crio-97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b WatchSource:0}: Error finding container 97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b: Status 404 returned error can't find the container with id 97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b
Oct 31 14:45:20.761717 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.761642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:20.761717 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.761711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:20.761883 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.761795    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:20.761883 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.761861    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:21.761847338 +0000 UTC m=+3.251886758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:20.761883 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.761795    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:20.761981 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.761913    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:21.761905123 +0000 UTC m=+3.251944544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:20.819978 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.819951    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:20.828474 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.828453    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:20.862206 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.862172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:20.862348 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.862329    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:20.862388 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.862350    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:20.862388 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.862360    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:20.862453 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:20.862414    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:21.862399526 +0000 UTC m=+3.352438963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:20.981651 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.981594    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp" event={"ID":"f4c37336-0737-4801-9213-08d878a4e6ad","Type":"ContainerStarted","Data":"fb4f70dd5023dbb6230463c444c208908906ab4a8bdd491bae43cdd4949b5582"}
Oct 31 14:45:20.982540 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.982519    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhqss" event={"ID":"7dc686b7-618e-47ee-b28e-35f1cbee4212","Type":"ContainerStarted","Data":"7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3"}
Oct 31 14:45:20.983605 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.983570    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktm59" event={"ID":"ebed5d09-55bf-4935-8ea9-55360332d683","Type":"ContainerStarted","Data":"e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b"}
Oct 31 14:45:20.984610 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.984584    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerStarted","Data":"085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb"}
Oct 31 14:45:20.985599 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.985565    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"d57afef5546aa1919f861928dcb50ce6","Type":"ContainerStarted","Data":"45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515"}
Oct 31 14:45:20.986665 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.986643    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s2smk" event={"ID":"381c174e-6f38-4afb-a8f1-08514439fa50","Type":"ContainerStarted","Data":"9ed339d3a6048e229999a4ca0cfe291475f00f1980343ae2bd9751f5ad0b32e2"}
Oct 31 14:45:20.987736 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.987711    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b"}
Oct 31 14:45:20.988735 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:20.988714    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ccjhc" event={"ID":"8120bc72-1aec-417f-9be2-4bc93ef332b5","Type":"ContainerStarted","Data":"d1bbabbcfb0fc046ff427868183ab054b3b663bb434f147b8a4c4265b014c5df"}
Oct 31 14:45:21.248666 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.248593    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:20 +0000 UTC" deadline="2028-05-14 07:14:33.999243168 +0000 UTC"
Oct 31 14:45:21.248666 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.248658    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h29m12.750587602s"
Oct 31 14:45:21.344164 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:21.344115    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecc9d82_e0fe_40a1_9bf2_c924e81b4433.slice/crio-747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907 WatchSource:0}: Error finding container 747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907: Status 404 returned error can't find the container with id 747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907
Oct 31 14:45:21.380362 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:21.380330    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e554f5_645d_42c9_b73d_8c1f8fa88d80.slice/crio-55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888 WatchSource:0}: Error finding container 55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888: Status 404 returned error can't find the container with id 55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888
Oct 31 14:45:21.381352 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:21.381327    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32e72b8_08d0_4824_9b97_305921ef0026.slice/crio-5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d WatchSource:0}: Error finding container 5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d: Status 404 returned error can't find the container with id 5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d
Oct 31 14:45:21.436158 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.436121    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:21.769877 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.769841    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:21.770064 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.769904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:21.770064 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.770057    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:21.770201 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.770120    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:23.770101951 +0000 UTC m=+5.260141391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:21.770201 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.770190    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:21.770321 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.770226    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:23.770213158 +0000 UTC m=+5.260252596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:21.870893 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.870854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:21.871453 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.871067    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:21.871453 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.871093    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:21.871453 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.871106    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:21.871453 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.871164    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:23.871146152 +0000 UTC m=+5.361185590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:21.977970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.977934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:21.978180 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.977976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:21.978180 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.978081    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:21.978180 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.978150    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:21.978180 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:21.978160    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:21.978401 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:21.978259    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:22.014428 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:22.014317    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerStarted","Data":"5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d"}
Oct 31 14:45:22.017483 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:22.017387    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjwxr" event={"ID":"97e554f5-645d-42c9-b73d-8c1f8fa88d80","Type":"ContainerStarted","Data":"55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888"}
Oct 31 14:45:22.030164 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:22.030081    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swww9" event={"ID":"aecc9d82-e0fe-40a1-9bf2-c924e81b4433","Type":"ContainerStarted","Data":"747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907"}
Oct 31 14:45:22.249545 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:22.249492    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:20 +0000 UTC" deadline="2028-05-01 21:13:11.347529944 +0000 UTC"
Oct 31 14:45:22.249545 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:22.249542    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h27m49.097993824s"
Oct 31 14:45:23.789214 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.789156    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:23.789834 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.789383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:23.789834 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.789484    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:23.789834 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.789549    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:27.78952998 +0000 UTC m=+9.279569418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:23.789834 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.789565    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:23.789834 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.789599    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:27.789588582 +0000 UTC m=+9.279628022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:23.890701 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.890301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:23.890701 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.890507    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:23.890701 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.890532    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:23.890701 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.890545    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:23.890701 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.890623    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:27.890587848 +0000 UTC m=+9.380627289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:23.979150 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.979013    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:23.979334 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.979169    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:23.979334 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.979249    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:23.979334 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.979322    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:23.979572 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:23.979369    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:23.979572 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:23.979442    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:25.978049 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:25.977990    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:25.978493 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:25.978139    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:25.978493 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:25.978197    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:25.978493 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:25.978268    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:25.978493 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:25.978303    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:25.978493 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:25.978359    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:26.091082 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:26.090815    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktm59" event={"ID":"ebed5d09-55bf-4935-8ea9-55360332d683","Type":"ContainerStarted","Data":"8a1b359af3f57bc386949c80b0caec18eda9ae693cbdcc07efe8d18fbbd3bcf1"}
Oct 31 14:45:26.101308 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:26.101239    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerStarted","Data":"9d9426afe682cb02d4bbb029e8b9dcd90860495be6336d3b2c50c4e4e00d3f30"}
Oct 31 14:45:26.215311 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:26.215229    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ktm59" podStartSLOduration=1.237712425 podStartE2EDuration="6.215201793s" totalImagesPullingTime="4.977489368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:26.213577624 +0000 UTC m=+7.703617074" watchObservedRunningTime="2026-10-31 14:45:26.215201793 +0000 UTC m=+7.705241242"
Oct 31 14:45:27.105954 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.105908    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b1c519a9e746dec2e3a5ff69d67b2ad" containerID="9685e69bebc5487c0a72b5d576cfd25d61904fd29fd777e01b3899eb6f3c2ab9" exitCode=0
Oct 31 14:45:27.106455 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.106011    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerDied","Data":"9685e69bebc5487c0a72b5d576cfd25d61904fd29fd777e01b3899eb6f3c2ab9"}
Oct 31 14:45:27.110970 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.110423    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"d57afef5546aa1919f861928dcb50ce6","Type":"ContainerStarted","Data":"a371bd503cce52b280f746794adc865dcc083203e9ddb201e6e8bdb515a63f5d"}
Oct 31 14:45:27.112910 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.112883    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="9d9426afe682cb02d4bbb029e8b9dcd90860495be6336d3b2c50c4e4e00d3f30" exitCode=0
Oct 31 14:45:27.113370 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.113347    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"9d9426afe682cb02d4bbb029e8b9dcd90860495be6336d3b2c50c4e4e00d3f30"}
Oct 31 14:45:27.830934 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.830893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:27.831151 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.830994    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:27.831151 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.831123    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:27.831287 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.831188    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:35.831169132 +0000 UTC m=+17.321208571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:27.831287 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.831258    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:27.831287 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.831294    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:35.831281294 +0000 UTC m=+17.321320722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:27.931852 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.931756    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:27.932016 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.931974    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:27.932016 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.931995    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:27.932016 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.932007    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:27.932177 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.932067    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:35.932047339 +0000 UTC m=+17.422086764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:27.978005 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.977969    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:27.978180 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.978114    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:27.978254 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.978195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:27.978308 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.978265    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:27.978365 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:27.978335    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:27.978446 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:27.978425    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:28.119692 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.119628    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp" event={"ID":"f4c37336-0737-4801-9213-08d878a4e6ad","Type":"ContainerStarted","Data":"caa7844404a49de9efc1cf5aee3e648b910c5580c95f2299beb0764ecbb17736"}
Oct 31 14:45:28.122718 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.122684    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gjwxr" event={"ID":"97e554f5-645d-42c9-b73d-8c1f8fa88d80","Type":"ContainerStarted","Data":"c5d18509e4c7963f6540c477cc8a9e356b4a33975a90b38a36506d1f10102191"}
Oct 31 14:45:28.128147 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.128111    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerStarted","Data":"5837b78d01ace63d2c90cc8ec5b5df40047c7d6b8b18fbdabf461aa0d9753560"}
Oct 31 14:45:28.245563 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.245427    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal" podStartSLOduration=8.245408501 podStartE2EDuration="8.245408501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:27.480145937 +0000 UTC m=+8.970185387" watchObservedRunningTime="2026-10-31 14:45:28.245408501 +0000 UTC m=+9.735447949"
Oct 31 14:45:28.367125 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.367057    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gjwxr" podStartSLOduration=2.2260211229999998 podStartE2EDuration="8.367036011s" totalImagesPullingTime="6.141014888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:28.245020869 +0000 UTC m=+9.735060329" watchObservedRunningTime="2026-10-31 14:45:28.367036011 +0000 UTC m=+9.857075459"
Oct 31 14:45:28.484686 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.484302    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:28.485639 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.485613    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:28.607624 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:28.607457    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" podStartSLOduration=8.607434904 podStartE2EDuration="8.607434904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:28.367725119 +0000 UTC m=+9.857764569" watchObservedRunningTime="2026-10-31 14:45:28.607434904 +0000 UTC m=+10.097474363"
Oct 31 14:45:29.517267 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.517234    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 14:45:29.916937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.915309    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:29.517262484Z","UUID":"e2bf1e54-b6aa-45f7-8822-c621f2aed6bf","Handler":null,"Name":"","Endpoint":""}
Oct 31 14:45:29.917733 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.917261    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 14:45:29.917733 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.917294    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 14:45:29.978733 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.978701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:29.978872 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:29.978827    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:29.978938 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.978896    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:29.978991 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:29.978979    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:29.979054 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:29.979041    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:29.979150 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:29.979126    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:30.135289 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:30.135250    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp" event={"ID":"f4c37336-0737-4801-9213-08d878a4e6ad","Type":"ContainerStarted","Data":"cc120919623e9a650d45d924cd6cb522142170311e1ba1ef04e7197e04e71f71"}
Oct 31 14:45:30.138769 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:30.138738    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bhqss" event={"ID":"7dc686b7-618e-47ee-b28e-35f1cbee4212","Type":"ContainerStarted","Data":"1528f578a35f7c945f365638c506a3127c077170f6922a26128c35a56b945d4b"}
Oct 31 14:45:31.144313 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:31.143847    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swww9" event={"ID":"aecc9d82-e0fe-40a1-9bf2-c924e81b4433","Type":"ContainerStarted","Data":"451d46d18c0a96f24ab7604969a8ca503f02f0e3ad26d709cd2c1a48009c5085"}
Oct 31 14:45:31.272312 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:31.272252    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bhqss" podStartSLOduration=1.997174709 podStartE2EDuration="11.272233074s" totalImagesPullingTime="9.275058365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:30.26122967 +0000 UTC m=+11.751269118" watchObservedRunningTime="2026-10-31 14:45:31.272233074 +0000 UTC m=+12.762272523"
Oct 31 14:45:31.977840 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:31.977564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:31.978010 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:31.977878    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:31.978010 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:31.977739    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:31.978010 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:31.977957    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:31.978010 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:31.977650    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:31.978223 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:31.978057    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:32.149620 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.149561    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp" event={"ID":"f4c37336-0737-4801-9213-08d878a4e6ad","Type":"ContainerStarted","Data":"a9f14a75cf2c3ae387cf09bcc1e1f93a4be228b0a002773263ae98562e173b54"}
Oct 31 14:45:32.151374 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.151344    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ccjhc" event={"ID":"8120bc72-1aec-417f-9be2-4bc93ef332b5","Type":"ContainerStarted","Data":"5f5d6f516c02ec26a3a7d91c3d096ce6dc9a2198df362831aa09759ad1218a98"}
Oct 31 14:45:32.154381 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.154347    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="4d4109138569a23c5e91310cd68856ab162f5cfd22f241a12d64580d9fceb672" exitCode=0
Oct 31 14:45:32.154497 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.154434    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"4d4109138569a23c5e91310cd68856ab162f5cfd22f241a12d64580d9fceb672"}
Oct 31 14:45:32.263152 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.263070    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp" podStartSLOduration=2.040386641 podStartE2EDuration="12.263056501s" totalImagesPullingTime="10.22266986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:32.262562098 +0000 UTC m=+13.752601543" watchObservedRunningTime="2026-10-31 14:45:32.263056501 +0000 UTC m=+13.753095948"
Oct 31 14:45:32.263300 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.263262    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-swww9" podStartSLOduration=4.25468721 podStartE2EDuration="13.263256966s" totalImagesPullingTime="9.008569756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:31.272583729 +0000 UTC m=+12.762623179" watchObservedRunningTime="2026-10-31 14:45:32.263256966 +0000 UTC m=+13.753296409"
Oct 31 14:45:32.493539 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:32.493483    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ccjhc" podStartSLOduration=3.471553911 podStartE2EDuration="12.493462426s" totalImagesPullingTime="9.021908515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:32.373067235 +0000 UTC m=+13.863106679" watchObservedRunningTime="2026-10-31 14:45:32.493462426 +0000 UTC m=+13.983501873"
Oct 31 14:45:33.158027 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:33.157992    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s2smk" event={"ID":"381c174e-6f38-4afb-a8f1-08514439fa50","Type":"ContainerStarted","Data":"5afc06cad9925e0e1d38bb31936d16e85019db4eb168254a544941a609021f29"}
Oct 31 14:45:33.272206 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:33.272147    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s2smk" podStartSLOduration=1.458868168 podStartE2EDuration="13.272126753s" totalImagesPullingTime="11.813258585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:33.271368772 +0000 UTC m=+14.761408216" watchObservedRunningTime="2026-10-31 14:45:33.272126753 +0000 UTC m=+14.762166197"
Oct 31 14:45:33.978427 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:33.978393    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:33.978427 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:33.978417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:33.978662 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:33.978520    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:33.978662 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:33.978618    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:33.978744 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:33.978686    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:33.978774 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:33.978741    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:34.163387 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:34.163344    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="fbb6c644ba67676fbf7917a515b878989b34bb1362755de9fafe5a1b2de2fc12" exitCode=0
Oct 31 14:45:34.164145 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:34.163425    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"fbb6c644ba67676fbf7917a515b878989b34bb1362755de9fafe5a1b2de2fc12"}
Oct 31 14:45:35.896723 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.896507    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:35.897215 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.896773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:35.897215 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.896680    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:35.897215 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.896871    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret podName:32080cb0-02b5-44e5-a7b2-3f33c5f97f48 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.896849615 +0000 UTC m=+33.386889056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret") pod "global-pull-secret-syncer-hmc4m" (UID: "32080cb0-02b5-44e5-a7b2-3f33c5f97f48") : object "kube-system"/"original-pull-secret" not registered
Oct 31 14:45:35.897215 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.896872    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:35.897215 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.896923    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.896906873 +0000 UTC m=+33.386946296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:35.977840 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.977809    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:35.978005 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.977916    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:35.978005 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.977985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:35.978079 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.978044    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:35.978134 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.978094    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:35.978204 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.978184    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:35.998095 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:35.998052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:35.998238 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.998217    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 14:45:35.998294 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.998240    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 14:45:35.998294 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.998250    2572 projected.go:196] Error preparing data for projected volume kube-api-access-2w6lm for pod openshift-network-diagnostics/network-check-target-ljbk9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:35.998397 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:35.998302    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm podName:a28abdca-0d74-42d8-b328-520e11f32600 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.998283328 +0000 UTC m=+33.488322751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2w6lm" (UniqueName: "kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm") pod "network-check-target-ljbk9" (UID: "a28abdca-0d74-42d8-b328-520e11f32600") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 14:45:36.171165 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:36.171084    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="4ea71fc251db50af247365a56f1820d7cf8d8d66286f6d6906b911f5d74d19f0" exitCode=0
Oct 31 14:45:36.171165 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:36.171115    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"4ea71fc251db50af247365a56f1820d7cf8d8d66286f6d6906b911f5d74d19f0"}
Oct 31 14:45:37.978610 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:37.978368    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:37.978610 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:37.978445    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:37.978610 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:37.978454    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:37.979178 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:37.978664    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:37.979178 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:37.978742    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:37.979178 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:37.978845    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:38.178730 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178704    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"cc9f0a2dee60fed20d45f1e77f71df50009951be118e3db6faab0768baa7b9ce"}
Oct 31 14:45:38.178837 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178769    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"4b2591c65f888c9efd18c24f9963a3c50171ea14f56e3ff2844be406aaded5b2"}
Oct 31 14:45:38.178837 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178798    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"ba47cfd800e2f78b90cb232736ca9908e2f1d0b22a7aec009f171ea62cd2a77c"}
Oct 31 14:45:38.178837 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178825    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"247e2eeabc494cb8a98d75731829000cb2029eb1af5ec9ee78d4db338aa97bc2"}
Oct 31 14:45:38.178929 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178851    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"d1c733ad03bfd3d61ee2472516ad288ca781ce77c2c9ac2884de6a4ea58e0b62"}
Oct 31 14:45:38.178929 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:38.178880    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"17c7cc850d3d0c1e18744030bf797ff0f155e1a3243a15c57bc337141c5d4381"}
Oct 31 14:45:39.183478 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:39.183443    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="61cb5e680eae8790504760658411cc01096d4c5bfa453c756abfb965bcf44411" exitCode=0
Oct 31 14:45:39.183877 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:39.183485    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"61cb5e680eae8790504760658411cc01096d4c5bfa453c756abfb965bcf44411"}
Oct 31 14:45:39.977916 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:39.977888    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:39.978044 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:39.977954    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:39.978107 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:39.978052    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:39.978107 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:39.978046    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:39.978183 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:39.978120    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:39.978224 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:39.978214    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:40.188493 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:40.188403    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"f18d64e854bb8ed37ace05d2213740a7ddf61d80624b381e95c6874bcc75ff5e"}
Oct 31 14:45:40.191179 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:40.191152    2572 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="8285a082840549c2d4fca7ff6ba60ded0b9cd9f8d1e56b45daa4a4e1106c4644" exitCode=0
Oct 31 14:45:40.191286 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:40.191183    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerDied","Data":"8285a082840549c2d4fca7ff6ba60ded0b9cd9f8d1e56b45daa4a4e1106c4644"}
Oct 31 14:45:41.196045 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:41.196012    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" event={"ID":"a32e72b8-08d0-4824-9b97-305921ef0026","Type":"ContainerStarted","Data":"b589aa119674cca9452b5645268a77dff3bf1dd042bead9827f46cfc028f0a5a"}
Oct 31 14:45:41.411846 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:41.411788    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lnk8v" podStartSLOduration=8.646800116 podStartE2EDuration="21.411774247s" totalImagesPullingTime="12.764974131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:41.361778152 +0000 UTC m=+22.851817596" watchObservedRunningTime="2026-10-31 14:45:41.411774247 +0000 UTC m=+22.901813689"
Oct 31 14:45:41.977564 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:41.977533    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:41.977746 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:41.977630    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:41.977746 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:41.977721    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:41.977746 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:41.977732    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:41.977867 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:41.977820    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:41.977914 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:41.977896    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:42.201766 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:42.201676    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" event={"ID":"f63e0003-a810-483a-8ab0-4e01ea21f4e9","Type":"ContainerStarted","Data":"a7c46a08e1bd95f3636184fce80715eca9b580af3b4ba81a945198d6979a9e31"}
Oct 31 14:45:42.432735 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:42.432684    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm" podStartSLOduration=5.572966956 podStartE2EDuration="22.432669027s" totalImagesPullingTime="16.859702071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:42.383397104 +0000 UTC m=+23.873436546" watchObservedRunningTime="2026-10-31 14:45:42.432669027 +0000 UTC m=+23.922708469"
Oct 31 14:45:43.204227 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.204196    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:43.204910 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.204281    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:43.204910 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.204329    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:43.219308 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.219239    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:43.219486 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.219471    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:45:43.977857 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.977829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:43.977857 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.977864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:43.978062 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:43.977927    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:43.978062 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:43.977998    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:43.978127 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:43.978070    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:43.978127 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:43.978113    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:44.664224 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.663985    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljbk9"]
Oct 31 14:45:44.664655 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.664302    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:44.664655 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:44.664395    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:44.667879 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.667856    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hmc4m"]
Oct 31 14:45:44.668015 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.667949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:44.668067 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:44.668045    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:44.668494 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.668473    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-64bfw"]
Oct 31 14:45:44.668601 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:44.668564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:44.668699 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:44.668682    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:45.977856 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:45.977829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:45.978377 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:45.977923    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:45.978377 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:45.978002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:45.978377 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:45.978095    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:46.978261 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:46.978188    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:46.978658 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:46.978293    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:47.366451 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:47.366417    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:47.366675 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:47.366548    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 14:45:47.366993 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:47.366980    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:45:47.977695 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:47.977662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:47.977695 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:47.977686    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:47.977930 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:47.977774    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:47.977930 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:47.977901    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:48.978400 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:48.978373    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:48.978828 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:48.978452    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hmc4m" podUID="32080cb0-02b5-44e5-a7b2-3f33c5f97f48"
Oct 31 14:45:49.977784 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:49.977748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:49.977969 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:49.977793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:49.977969 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:49.977864    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-64bfw" podUID="309034fe-5db8-476f-b335-41a37a2c9464"
Oct 31 14:45:49.978058 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:49.977969    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ljbk9" podUID="a28abdca-0d74-42d8-b328-520e11f32600"
Oct 31 14:45:50.256536 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.256468    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeReady"
Oct 31 14:45:50.256927 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.256630    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 14:45:50.357759 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.357729    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ls8dq"]
Oct 31 14:45:50.361205 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.361189    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.366217 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.366184    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jk8rn"]
Oct 31 14:45:50.369800 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.369784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.378137 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.378115    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cblh8"]
Oct 31 14:45:50.380937 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.380921    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.381217 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.381197    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:45:50.384165 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.384149    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.418707 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.418668    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ls8dq"]
Oct 31 14:45:50.432666 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432634    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 14:45:50.432812 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432713    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 14:45:50.432812 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432721    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 14:45:50.432812 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432802    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\""
Oct 31 14:45:50.432962 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432850    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 14:45:50.432962 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432860    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 14:45:50.432962 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432898    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 14:45:50.432962 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432939    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 14:45:50.433137 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.432992    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 14:45:50.433137 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.433034    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 14:45:50.433137 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.433111    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\""
Oct 31 14:45:50.433137 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.433137    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\""
Oct 31 14:45:50.438113 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.438090    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cblh8"]
Oct 31 14:45:50.446107 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.446083    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:45:50.483103 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.483066    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk8rn"]
Oct 31 14:45:50.507158 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.507158 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507119    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.507158 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.507354 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507199    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.507354 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h755\" (UniqueName: \"kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.507354 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.507354 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.507468 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.507468 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.507523 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvbs7\" (UniqueName: \"kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.507523 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507510    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.507613 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507530    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.507613 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.507613 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.507695 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.507641    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddg5\" (UniqueName: \"kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.531413 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.531383    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 14:45:50.531531 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.531509    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\""
Oct 31 14:45:50.581239 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.581204    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 14:45:50.608729 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.608729 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h755\" (UniqueName: \"kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608780    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.608841    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.608916    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.108895425 +0000 UTC m=+32.598934867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608940    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbs7\" (UniqueName: \"kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.608972 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608972    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.608997    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609016    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609037    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609066    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.609092    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddg5\" (UniqueName: \"kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.609142    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.10912697 +0000 UTC m=+32.599166408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609232    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609272    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.609399 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.609391    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:50.609440    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.109426614 +0000 UTC m=+32.599466035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609446    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609564    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609692    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609761    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.609947 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.609916    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.612909 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.612891    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.665789 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.665756    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddg5\" (UniqueName: \"kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:50.669158 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.669123    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h755\" (UniqueName: \"kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:50.669457 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.669436    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.669610 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.669560    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbs7\" (UniqueName: \"kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.691745 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.691720    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:45:50.696861 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.696842    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:45:50.978041 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:50.978011    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:51.030769 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.030740    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 14:45:51.030769 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.030743    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\""
Oct 31 14:45:51.082966 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.082942    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cblh8"]
Oct 31 14:45:51.086238 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:51.086215    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a902571_08a4_48f8_9384_caed28d8893e.slice/crio-e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0 WatchSource:0}: Error finding container e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0: Status 404 returned error can't find the container with id e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0
Oct 31 14:45:51.113703 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.113676    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:51.113830 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.113721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:51.113830 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.113744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:51.113830 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113811    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:45:51.113963 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113851    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:45:51.113963 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113849    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:51.113963 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113861    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:52.113849066 +0000 UTC m=+33.603888487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:45:51.113963 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113945    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:52.113931944 +0000 UTC m=+33.603971365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:45:51.113963 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.113956    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:52.113949778 +0000 UTC m=+33.603989199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:51.129537 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.129512    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:45:51.132658 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:51.132635    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bd6a76_b8c7_45e9_8d16_678fd598eecb.slice/crio-c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47 WatchSource:0}: Error finding container c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47: Status 404 returned error can't find the container with id c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47
Oct 31 14:45:51.185470 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.185439    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"]
Oct 31 14:45:51.219736 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.219708    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:45:51.224939 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.224907    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cblh8" event={"ID":"5a902571-08a4-48f8-9384-caed28d8893e","Type":"ContainerStarted","Data":"e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0"}
Oct 31 14:45:51.226147 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.226074    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" event={"ID":"c1bd6a76-b8c7-45e9-8d16-678fd598eecb","Type":"ContainerStarted","Data":"c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47"}
Oct 31 14:45:51.244048 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.244009    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"]
Oct 31 14:45:51.315555 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.315470    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjmdb\" (UniqueName: \"kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb\") pod \"migrator-66df5d496b-t42s8\" (UID: \"1c844afc-670c-45f5-82bb-9f53eb55ac91\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:45:51.347001 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.346925    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 14:45:51.347191 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.347145    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\""
Oct 31 14:45:51.347298 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.347280    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 14:45:51.416423 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.416387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmdb\" (UniqueName: \"kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb\") pod \"migrator-66df5d496b-t42s8\" (UID: \"1c844afc-670c-45f5-82bb-9f53eb55ac91\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:45:51.473094 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.473060    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmdb\" (UniqueName: \"kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb\") pod \"migrator-66df5d496b-t42s8\" (UID: \"1c844afc-670c-45f5-82bb-9f53eb55ac91\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:45:51.543288 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.543262    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:45:51.920329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.920246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:51.920329 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.920293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:51.920532 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.920399    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:51.920532 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:51.920462    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:23.92044704 +0000 UTC m=+65.410486475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 14:45:51.922665 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.922647    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:51.927402 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.927377    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"]
Oct 31 14:45:51.930207 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:51.930181    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c844afc_670c_45f5_82bb_9f53eb55ac91.slice/crio-0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37 WatchSource:0}: Error finding container 0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37: Status 404 returned error can't find the container with id 0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37
Oct 31 14:45:51.978001 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.977964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:45:51.978001 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:51.977994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:52.021653 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.021613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:52.030176 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.030144    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 14:45:52.030176 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.030156    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 14:45:52.030332 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.030184    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 14:45:52.030332 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.030228    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\""
Oct 31 14:45:52.030402 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.030357    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\""
Oct 31 14:45:52.035175 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.035152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.122368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.122438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.122460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.122560    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.122632    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:54.122613785 +0000 UTC m=+35.612653221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.123308    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.123360    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:54.123345378 +0000 UTC m=+35.613384814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.123637    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:45:52.123707 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:52.123671    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:54.123662828 +0000 UTC m=+35.613702262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:45:52.193440 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.193357    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:45:52.230308 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.230275    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8" event={"ID":"1c844afc-670c-45f5-82bb-9f53eb55ac91","Type":"ContainerStarted","Data":"0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37"}
Oct 31 14:45:52.328978 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.328931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:45:52.597088 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.597052    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hmc4m"]
Oct 31 14:45:52.601412 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:52.601387    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32080cb0_02b5_44e5_a7b2_3f33c5f97f48.slice/crio-f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063 WatchSource:0}: Error finding container f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063: Status 404 returned error can't find the container with id f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063
Oct 31 14:45:52.726004 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.725971    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ljbk9"]
Oct 31 14:45:52.732723 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:52.732689    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28abdca_0d74_42d8_b328_520e11f32600.slice/crio-eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3 WatchSource:0}: Error finding container eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3: Status 404 returned error can't find the container with id eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3
Oct 31 14:45:52.814013 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.813976    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26fwk"]
Oct 31 14:45:52.855041 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.854967    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:52.872603 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.872559    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26fwk"]
Oct 31 14:45:52.907656 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.907048    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 14:45:52.907656 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.907053    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 14:45:52.907656 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.907053    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 14:45:52.907656 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.907054    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 14:45:52.907656 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.907142    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j8ksf\""
Oct 31 14:45:52.929814 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.929610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:52.929814 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.929661    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:52.929814 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:52.929692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprcb\" (UniqueName: \"kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.030222 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.030177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.030409 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.030246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.030409 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.030345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprcb\" (UniqueName: \"kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.031036 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.031006    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.035362 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.034306    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.089605 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.088966    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprcb\" (UniqueName: \"kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.181530 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.181448    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:45:53.239105 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.239059    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hmc4m" event={"ID":"32080cb0-02b5-44e5-a7b2-3f33c5f97f48","Type":"ContainerStarted","Data":"f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063"}
Oct 31 14:45:53.243188 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.243149    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljbk9" event={"ID":"a28abdca-0d74-42d8-b328-520e11f32600","Type":"ContainerStarted","Data":"eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3"}
Oct 31 14:45:53.591042 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:53.591005    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-26fwk"]
Oct 31 14:45:53.594392 ip-10-0-139-115 kubenswrapper[2572]: W1031 14:45:53.594359    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d77bf10_10c6_445e_9c93_9832cc17ee67.slice/crio-3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e WatchSource:0}: Error finding container 3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e: Status 404 returned error can't find the container with id 3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e
Oct 31 14:45:54.145160 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.145120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:54.145318 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.145185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:54.145318 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.145223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:54.145318 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145291    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:54.145473 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145336    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:45:54.145473 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145339    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:45:54.145473 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145392    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.14537207 +0000 UTC m=+39.635411495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:54.145473 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145418    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.145399521 +0000 UTC m=+39.635438956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:45:54.145473 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:54.145436    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.145427334 +0000 UTC m=+39.635466761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:45:54.248286 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.248198    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8" event={"ID":"1c844afc-670c-45f5-82bb-9f53eb55ac91","Type":"ContainerStarted","Data":"b79a4c44b309138a92ec95ad6d0b389159d7dca1c14b27dce0f8dec77f79701b"}
Oct 31 14:45:54.249769 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.249738    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk" event={"ID":"3d77bf10-10c6-445e-9c93-9832cc17ee67","Type":"ContainerStarted","Data":"3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e"}
Oct 31 14:45:54.368318 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:54.368189    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8" podStartSLOduration=1.241598981 podStartE2EDuration="3.368168493s" totalImagesPullingTime="2.126569512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:54.367621546 +0000 UTC m=+35.857660990" watchObservedRunningTime="2026-10-31 14:45:54.368168493 +0000 UTC m=+35.858207937"
Oct 31 14:45:55.257549 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:55.257517    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cblh8" event={"ID":"5a902571-08a4-48f8-9384-caed28d8893e","Type":"ContainerStarted","Data":"d399e1ff104983cecba2b96098ea8b22f87c0252590d5fed61f8b1a23865d59c"}
Oct 31 14:45:55.261786 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:55.261755    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" event={"ID":"c1bd6a76-b8c7-45e9-8d16-678fd598eecb","Type":"ContainerStarted","Data":"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"}
Oct 31 14:45:55.375923 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:55.375864    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" podStartSLOduration=2.122257549 podStartE2EDuration="5.375846938s" totalImagesPullingTime="3.253589389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:55.374770059 +0000 UTC m=+36.864809502" watchObservedRunningTime="2026-10-31 14:45:55.375846938 +0000 UTC m=+36.865886380"
Oct 31 14:45:56.382742 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:56.382090    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cblh8" podStartSLOduration=2.313694348 podStartE2EDuration="6.382072407s" totalImagesPullingTime="4.068378059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:56.381527012 +0000 UTC m=+37.871566455" watchObservedRunningTime="2026-10-31 14:45:56.382072407 +0000 UTC m=+37.872111852"
Oct 31 14:45:57.268870 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:57.268827    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk" event={"ID":"3d77bf10-10c6-445e-9c93-9832cc17ee67","Type":"ContainerStarted","Data":"087b0ddc0d372bd8da72c84b913ef636bf89cb985340f9051a04e21b7fe6ed14"}
Oct 31 14:45:57.382003 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:57.381944    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk" podStartSLOduration=1.836701492 podStartE2EDuration="5.381926669s" totalImagesPullingTime="3.545225177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:57.380793404 +0000 UTC m=+38.870832852" watchObservedRunningTime="2026-10-31 14:45:57.381926669 +0000 UTC m=+38.871966115"
Oct 31 14:45:58.181072 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.181024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.181095    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.181134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181196    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181220    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181252    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181275    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:06.181253682 +0000 UTC m=+47.671293107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181296    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:06.181283421 +0000 UTC m=+47.671322842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:45:58.181595 ip-10-0-139-115 kubenswrapper[2572]: E1031 14:45:58.181312    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:06.181304279 +0000 UTC m=+47.671343701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:45:58.273806 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.273769    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ljbk9" event={"ID":"a28abdca-0d74-42d8-b328-520e11f32600","Type":"ContainerStarted","Data":"37c98c0623388ef26dae50a2fa25bb1d0cf102bb4a1bc08d851145a28ecbd933"}
Oct 31 14:45:58.308398 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.308371    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bhqss_7dc686b7-618e-47ee-b28e-35f1cbee4212/dns-node-resolver/0.log"
Oct 31 14:45:58.389371 ip-10-0-139-115 kubenswrapper[2572]: I1031 14:45:58.389317    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ljbk9" podStartSLOduration=33.73590885 podStartE2EDuration="38.389302476s" totalImagesPullingTime="4.653393626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:58.388744713 +0000 UTC m=+39.878784158" watchObservedRunningTime="2026-10-31 14:45:58.389302476 +0000 UTC m=+39.879341915"
Oct 31 14:45:59.164874 ip-10-0-139-115 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 14:45:59.176926 ip-10-0-139-115 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 14:45:59.177155 ip-10-0-139-115 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 14:45:59.177213 ip-10-0-139-115 systemd[1]: kubelet.service: Consumed 2.515s CPU time.
Oct 31 14:45:59.192316 ip-10-0-139-115 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 14:45:59.306461 ip-10-0-139-115 kubenswrapper[4549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:59.306461 ip-10-0-139-115 kubenswrapper[4549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 14:45:59.306461 ip-10-0-139-115 kubenswrapper[4549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:59.306461 ip-10-0-139-115 kubenswrapper[4549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 14:45:59.310014 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.309914    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.310014 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310016    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310022    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310026    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310031    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310036    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310040    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310044    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310049    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310053    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310057    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310061    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310065    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310069    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310073    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310077    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310081    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310085    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310092    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.310095 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310099    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310113    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310117    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310122    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310126    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310130    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310134    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310138    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310142    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310146    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310150    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310154    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310157    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310162    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310166    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310170    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310174    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310178    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310182    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.310547 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310186    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310190    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310194    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310198    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310202    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310206    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310213    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310218    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310223    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310227    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310232    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310237    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310240    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310245    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310250    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310254    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310258    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310262    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310266    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.311036 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310271    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310275    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310279    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310283    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310287    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310291    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310295    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310300    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310305    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310310    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310314    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310317    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310321    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310325    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310329    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310333    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310338    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310341    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310345    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310349    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.311528 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310353    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310357    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310361    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310366    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310370    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310374    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310378    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310383    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310387    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310391    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310395    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310399    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310403    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310406    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310410    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310414    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310418    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310422    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310426    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310429    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.312054 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310433    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310438    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310442    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310446    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310450    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310454    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310459    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310464    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.310468    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311659    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311667    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311671    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311676    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311679    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311683    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311689    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311693    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311697    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311701    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.312719 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311705    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311709    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311713    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311717    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311721    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311725    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311729    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311733    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311737    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311741    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311745    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311749    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311753    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311757    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311761    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311766    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311770    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311774    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311778    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311782    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.313212 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311788    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311792    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311796    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311800    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311804    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311808    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311811    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311816    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311820    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311824    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311828    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311832    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311836    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311840    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311844    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311848    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311852    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311856    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311860    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.313835 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311864    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311867    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311871    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311875    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311879    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311886    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311891    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311895    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311900    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311906    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311911    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311918    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311924    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311928    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311934    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311938    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311942    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311946    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311950    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.314317 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311954    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311958    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311962    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311966    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311970    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311974    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311978    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311982    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311985    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311989    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311993    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.311997    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312001    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312005    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312009    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312013    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312017    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312021    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312025    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312029    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.314812 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312033    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312037    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312095    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312101    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312105    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312109    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312113    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312117    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312122    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312126    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312130    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312134    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312138    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312142    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312146    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312149    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312154    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.312159    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313011    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.315296 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313033    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313038    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313054    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313058    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313062    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313067    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313071    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313075    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313080    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313084    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313088    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313092    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313103    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313107    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313111    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313116    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313120    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313125    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313129    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313133    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.315819 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313137    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313141    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313145    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313149    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313153    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313157    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313167    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313171    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313175    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313179    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313183    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313187    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313191    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313195    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313199    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313203    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313207    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313211    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313216    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.316403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313220    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313224    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313229    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313234    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313238    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313241    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313245    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313249    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313253    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313257    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313262    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313266    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313270    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313273    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313277    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313281    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313285    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313289    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313294    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.316910 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313297    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313301    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313307    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313313    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313319    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313325    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313331    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313339    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313343    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313348    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313352    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313357    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313361    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313367    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313371    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313375    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313379    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313383    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313387    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.317377 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313391    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313395    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313400    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313404    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313407    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313412    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313417    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313422    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313425    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313429    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313434    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313438    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313442    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313446    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313450    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313454    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313458    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313463    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313467    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313471    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.317938 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313479    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313483    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313488    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313494    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313498    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313502    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313506    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.313510    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313652    4549 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313662    4549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313670    4549 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313677    4549 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313683    4549 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313688    4549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313695    4549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313701    4549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313706    4549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313710    4549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313716    4549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313722    4549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313726    4549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313731    4549 flags.go:64] FLAG: --cgroup-root=""
Oct 31 14:45:59.318562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313735    4549 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313740    4549 flags.go:64] FLAG: --client-ca-file=""
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313744    4549 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313749    4549 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313755    4549 flags.go:64] FLAG: --cluster-domain=""
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313760    4549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313765    4549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313770    4549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313775    4549 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313781    4549 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313788    4549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313796    4549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313802    4549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313806    4549 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313811    4549 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313816    4549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313821    4549 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313825    4549 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313831    4549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313836    4549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313841    4549 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313845    4549 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313850    4549 flags.go:64] FLAG: --enable-server="true"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313855    4549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313861    4549 flags.go:64] FLAG: --event-burst="100"
Oct 31 14:45:59.319133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313866    4549 flags.go:64] FLAG: --event-qps="50"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313870    4549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313875    4549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313879    4549 flags.go:64] FLAG: --eviction-hard=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313886    4549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313891    4549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313896    4549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313901    4549 flags.go:64] FLAG: --eviction-soft=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313905    4549 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313910    4549 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313914    4549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313919    4549 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313923    4549 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313927    4549 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313932    4549 flags.go:64] FLAG: --feature-gates=""
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313938    4549 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313942    4549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313949    4549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313955    4549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313959    4549 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313964    4549 flags.go:64] FLAG: --help="false"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313969    4549 flags.go:64] FLAG: --hostname-override="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313974    4549 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313979    4549 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 14:45:59.319949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313983    4549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313988    4549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313994    4549 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.313998    4549 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314003    4549 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314008    4549 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314013    4549 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314017    4549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314022    4549 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314026    4549 flags.go:64] FLAG: --kube-reserved=""
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314031    4549 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314035    4549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314040    4549 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314045    4549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314049    4549 flags.go:64] FLAG: --lock-file=""
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314054    4549 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314058    4549 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314063    4549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314071    4549 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314076    4549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314080    4549 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314085    4549 flags.go:64] FLAG: --logging-format="text"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314089    4549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314094    4549 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 14:45:59.320795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314099    4549 flags.go:64] FLAG: --manifest-url=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314105    4549 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314114    4549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314118    4549 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314124    4549 flags.go:64] FLAG: --max-pods="110"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314129    4549 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314133    4549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314138    4549 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314142    4549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314146    4549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314151    4549 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314156    4549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314166    4549 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314171    4549 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314176    4549 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314181    4549 flags.go:64] FLAG: --pod-cidr=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314185    4549 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314189    4549 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314194    4549 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314198    4549 flags.go:64] FLAG: --port="10250"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314202    4549 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314207    4549 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-06ec943c9bfebf1d1"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314213    4549 flags.go:64] FLAG: --qos-reserved=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314217    4549 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314222    4549 flags.go:64] FLAG: --register-node="true"
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314227    4549 flags.go:64] FLAG: --register-with-taints=""
Oct 31 14:45:59.321400 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314232    4549 flags.go:64] FLAG: --registry-burst="10"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314237    4549 flags.go:64] FLAG: --registry-qps="5"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314241    4549 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314245    4549 flags.go:64] FLAG: --reserved-memory=""
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314251    4549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314255    4549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314260    4549 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314266    4549 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314272    4549 flags.go:64] FLAG: --runonce="false"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314276    4549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314281    4549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314286    4549 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314291    4549 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314295    4549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314300    4549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314305    4549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314310    4549 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314314    4549 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314318    4549 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314323    4549 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314327    4549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314332    4549 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314336    4549 flags.go:64] FLAG: --system-cgroups=""
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314341    4549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314349    4549 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 14:45:59.322069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314353    4549 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314357    4549 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314363    4549 flags.go:64] FLAG: --tls-min-version=""
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314368    4549 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314373    4549 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314377    4549 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314382    4549 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314386    4549 flags.go:64] FLAG: --v="2"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314393    4549 flags.go:64] FLAG: --version="false"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314399    4549 flags.go:64] FLAG: --vmodule=""
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314405    4549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.314410    4549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314652    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314658    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314663    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314668    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314672    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314676    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314683    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314688    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314693    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314698    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.322734 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314702    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314706    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314710    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314714    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314718    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314722    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314726    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314730    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314734    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314738    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314742    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314746    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314751    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314755    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314759    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314764    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314768    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314772    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314776    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314779    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.323359 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314784    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314788    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314792    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314796    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314800    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314804    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314808    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314813    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314817    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314821    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314825    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314829    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314833    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314837    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314841    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314845    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314849    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314852    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314856    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314860    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.324018 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314864    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314867    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314871    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314875    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314880    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314884    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314887    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314891    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314896    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314900    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314904    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314908    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314912    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314916    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314920    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314924    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314928    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314932    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314935    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.324559 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314939    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314943    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314947    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314951    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314955    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314959    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314963    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314967    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314971    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314975    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314980    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314984    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314988    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314992    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314995    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.314999    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315004    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315008    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315012    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315016    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.325281 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315021    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315025    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315029    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315033    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315037    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315041    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315047    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315053    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315058    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315064    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315068    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315072    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315076    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315080    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315084    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315088    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.325965 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315092    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.315129    4549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315182    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315189    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315194    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315198    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315202    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315206    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315210    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315214    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315218    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315221    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315227    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315233    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315237    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315241    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.326368 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315245    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315249    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315254    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315258    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315263    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315267    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315271    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315275    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315278    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315283    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315289    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315295    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315299    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315303    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315308    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315312    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315316    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315320    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315324    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.326878 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315328    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315332    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315336    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315340    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315344    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315348    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315352    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315356    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315362    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315368    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315374    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315379    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315383    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315387    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315391    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315394    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315399    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315405    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315409    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.327667 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315413    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315417    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315421    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315425    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315429    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315433    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315437    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315441    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315445    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315449    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315453    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315457    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315461    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315465    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315469    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315473    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315477    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315480    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315484    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315488    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.328307 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315492    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315496    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315502    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315507    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315511    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315515    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315519    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315523    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315527    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315531    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315542    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315546    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315550    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315554    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315558    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315562    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315582    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315586    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315590    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315594    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.328864 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315598    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315602    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315608    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315612    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315616    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315620    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315624    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315629    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315633    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315637    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315641    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315644    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315648    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.315652    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.321738    4549 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.321757    4549 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.321777    4549 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.321782    4549 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321846    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.329361 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321850    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321853    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321857    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321860    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321863    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321866    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321869    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321872    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321874    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321877    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321880    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321883    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321885    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321889    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321893    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321896    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321898    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321901    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.330064 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321903    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321906    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321909    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321912    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321915    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321917    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321920    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321922    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321925    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321928    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321931    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321934    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321936    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321939    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321941    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321945    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321949    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321952    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321955    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.330727 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321958    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321960    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321963    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321966    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321968    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321971    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321973    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321976    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321979    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321981    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321984    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321986    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321989    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321992    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321994    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321997    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.321999    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322001    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322004    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322007    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.331311 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322009    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322012    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322015    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322017    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322020    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322022    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322024    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322027    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322029    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322032    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322034    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322037    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322039    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322042    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322045    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322047    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322050    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322052    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322055    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322057    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.331849 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322060    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322063    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322066    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322068    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322071    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322073    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322076    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322079    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322082    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322084    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322088    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322090    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322093    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322096    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322099    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322101    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322104    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322106    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322108    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322111    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.332340 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322113    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322116    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322118    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322121    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322123    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322126    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322128    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322131    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.322160    4549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322318    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322323    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322326    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322329    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322331    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322334    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322338    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.332906 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322342    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322345    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322348    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322351    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322354    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322356    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322359    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322362    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322365    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322367    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322370    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322372    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322375    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322377    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322380    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322382    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322385    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322387    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322390    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322393    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.333328 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322395    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322397    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322400    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322403    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322405    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322408    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322410    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322412    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322415    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322418    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322420    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322426    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322429    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322432    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322435    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322437    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322440    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322443    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322445    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322448    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.333853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322450    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322453    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322455    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322458    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322460    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322463    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322465    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322468    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322471    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322473    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322475    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322478    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322480    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322483    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322485    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322488    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322490    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322493    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322495    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.334343 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322498    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322500    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322503    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322505    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322509    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322513    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322516    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322518    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322521    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322524    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322527    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322529    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322532    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322534    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322537    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322539    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322542    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322544    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322547    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.334872 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322549    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322552    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322556    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322559    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322562    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322565    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322584    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322587    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322589    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322592    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322595    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322597    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322600    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322603    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322605    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322608    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322610    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322614    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322617    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322620    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.335331 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322623    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.322649    4549 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322681    4549 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322691    4549 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322694    4549 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322697    4549 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322700    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322703    4549 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322705    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322708    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322710    4549 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322713    4549 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322715    4549 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322718    4549 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322720    4549 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322723    4549 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 14:45:59.335871 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322725    4549 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322728    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322731    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322733    4549 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322736    4549 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322738    4549 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322741    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322744    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322746    4549 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322752    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322754    4549 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322757    4549 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322759    4549 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322762    4549 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322766    4549 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322769    4549 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322774    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322777    4549 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322780    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 14:45:59.336391 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322782    4549 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322785    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322788    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322791    4549 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322793    4549 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322796    4549 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322799    4549 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322801    4549 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322804    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322806    4549 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322809    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322811    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322814    4549 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322818    4549 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322821    4549 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322824    4549 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322827    4549 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322830    4549 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322832    4549 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 14:45:59.336969 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322835    4549 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322837    4549 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322840    4549 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322845    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322847    4549 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322850    4549 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322853    4549 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322855    4549 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322859    4549 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322862    4549 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322865    4549 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322869    4549 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322871    4549 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322874    4549 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322876    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322879    4549 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322881    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322884    4549 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322886    4549 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322890    4549 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 14:45:59.337452 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322894    4549 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322898    4549 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322900    4549 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322903    4549 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322906    4549 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322909    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322911    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322914    4549 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322916    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322919    4549 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322922    4549 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322924    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322927    4549 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322929    4549 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322932    4549 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322936    4549 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322939    4549 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322941    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322943    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322946    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 14:45:59.338118 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322949    4549 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322952    4549 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322956    4549 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322959    4549 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322962    4549 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322965    4549 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322967    4549 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322970    4549 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322972    4549 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322975    4549 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322977    4549 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322980    4549 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322982    4549 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:45:59.322985    4549 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.323048    4549 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324221    4549 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324287    4549 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324468    4549 server.go:1014] "Starting client certificate rotation"
Oct 31 14:45:59.338643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324601    4549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324679    4549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:19 +0000 UTC" deadline="2028-06-03 08:27:06.646157411 +0000 UTC"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324720    4549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13937h41m7.321441134s"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.324942    4549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.326027    4549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.327705    4549 log.go:25] "Validated CRI v1 runtime API"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.329742    4549 log.go:25] "Validated CRI v1 image API"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.330542    4549 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 14:45:59.339302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.332783    4549 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a4b8cee-0da1-4113-ae4a-384be7747e44:/dev/nvme0n1p4 88e7f930-5261-495b-9f42-565a6aeb4a84:/dev/nvme0n1p3]
Oct 31 14:45:59.339710 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.332810    4549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1bbabbcfb0fc046ff427868183ab054b3b663bb434f147b8a4c4265b014c5df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1bbabbcfb0fc046ff427868183ab054b3b663bb434f147b8a4c4265b014c5df/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb4f70dd5023dbb6230463c444c208908906ab4a8bdd491bae43cdd4949b5582/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb4f70dd5023dbb6230463c444c208908906ab4a8bdd491bae43cdd4949b5582/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c844afc-670c-45f5-82bb-9f53eb55ac91/volumes/kubernetes.io~projected/kube-api-access-cjmdb:{mountpoint:/var/lib/kubelet/pods/1c844afc-670c-45f5-82bb-9f53eb55ac91/volumes/kubernetes.io~projected/kube-api-access-cjmdb major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/309034fe-5db8-476f-b335-41a37a2c9464/volumes/kubernetes.io~projected/kube-api-access-7hpkn:{mountpoint:/var/lib/kubelet/pods/309034fe-5db8-476f-b335-41a37a2c9464/volumes/kubernetes.io~projected/kube-api-access-7hpkn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32080cb0-02b5-44e5-a7b2-3f33c5f97f48/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/32080cb0-02b5-44e5-a7b2-3f33c5f97f48/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~projected/kube-api-access-62kcv:{mountpoint:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~projected/kube-api-access-62kcv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~projected/kube-api-access-sprcb:{mountpoint:/var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~projected/kube-api-access-sprcb major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~secret/signing-key major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~projected/kube-api-access-zvbs7:{mountpoint:/var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~projected/kube-api-access-zvbs7 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dc686b7-618e-47ee-b28e-35f1cbee4212/volumes/kubernetes.io~projected/kube-api-access-fmb6c:{mountpoint:/var/lib/kubelet/pods/7dc686b7-618e-47ee-b28e-35f1cbee4212/volumes/kubernetes.io~projected/kube-api-access-fmb6c major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8120bc72-1aec-417f-9be2-4bc93ef332b5/volumes/kubernetes.io~projected/kube-api-access-kqpnm:{mountpoint:/var/lib/kubelet/pods/8120bc72-1aec-417f-9be2-4bc93ef332b5/volumes/kubernetes.io~projected/kube-api-access-kqpnm major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97e554f5-645d-42c9-b73d-8c1f8fa88d80/volumes/kubernetes.io~projected/kube-api-access-qm2c8:{mountpoint:/var/lib/kubelet/pods/97e554f5-645d-42c9-b73d-8c1f8fa88d80/volumes/kubernetes.io~projected/kube-api-access-qm2c8 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a28abdca-0d74-42d8-b328-520e11f32600/volumes/kubernetes.io~projected/kube-api-access-2w6lm:{mountpoint:/var/lib/kubelet/pods/a28abdca-0d74-42d8-b328-520e11f32600/volumes/kubernetes.io~projected/kube-api-access-2w6lm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a32e72b8-08d0-4824-9b97-305921ef0026/volumes/kubernetes.io~projected/kube-api-access-zf6tx:{mountpoint:/var/lib/kubelet/pods/a32e72b8-08d0-4824-9b97-305921ef0026/volumes/kubernetes.io~projected/kube-api-access-zf6tx major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aecc9d82-e0fe-40a1-9bf2-c924e81b4433/volumes/kubernetes.io~projected/kube-api-access-m9jwm:{mountpoint:/var/lib/kubelet/pods/aecc9d82-e0fe-40a1-9bf2-c924e81b4433/volumes/kubernetes.io~projected/kube-api-access-m9jwm major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bd6a76-b8c7-45e9-8d16-678fd598eecb/volumes/kubernetes.io~projected/kube-api-access-wjfhd:{mountpoint:/var/lib/kubelet/pods/c1bd6a76-b8c7-45e9-8d16-678fd598eecb/volumes/kubernetes.io~projected/kube-api-access-wjfhd major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d04e5de2-0045-40dc-b5cf-99533e3724c2/volumes/kubernetes.io~projected/kube-api-access-zddg5:{mountpoint:/var/lib/kubelet/pods/d04e5de2-0045-40dc-b5cf-99533e3724c2/volumes/kubernetes.io~projected/kube-api-access-zddg5 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e42ce53e-cac9-470c-aa88-cd889695d7c3/volumes/kubernetes.io~projected/kube-api-access-6h755:{mountpoint:/var/lib/kubelet/pods/e42ce53e-cac9-470c-aa88-cd889695d7c3/volumes/kubernetes.io~projected/kube-api-access-6h755 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebed5d09-55bf-4935-8ea9-55360332d683/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ebed5d09-55bf-4935-8ea9-55360332d683/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4c37336-0737-4801-9213-08d878a4e6ad/volumes/kubernetes.io~projected/kube-api-access-84zqg:{mountpoint:/var/lib/kubelet/pods/f4c37336-0737-4801-9213-08d878a4e6ad/volumes/kubernetes.io~projected/kube-api-access-84zqg major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~projected/kube-api-access-28t9l:{mountpoint:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~projected/kube-api-access-28t9l major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b60465403b04d443b37c80e00744532b41e7519f0bbaf2d9a2506cc3d59a015c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ba6bfbf60f294b8ccb7667e5e774eadb96ae54db1d3603c64ea6247e3807f9d2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a916bf75e054b7f6cae5a30cda5b96fc525daf5f7f1f80174f322c5d6fa9e944/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c5d9dd7f7957c4ccd2236336949783b20709803832d69c6d2e2daf5bc5875cc8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/8e65ea76fe94a80af19bec8ab88e3f54afde11c8b7edbbc85237517430bf57d5/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/df5de70a656ed1d01af0e639964a85464d365a9c1f63abed134645ee70d9ed7d/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/f7ac604590f829ba67839fc1dec0758f734b3b69a5d679c92db43002fe43baf5/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9f81afaa89507eff4dc7418915200e416bfd38ae4b5b485ad5a88a8631411a1e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a636856139293cac8b079e2198a157bce13cdfa73f6177e9f60b4198d2ee4d4b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d42f154629ff36a1d56c07f8ef9dddf1f51a0af96a46850bd97e4c956516752a/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/8a5b832067c57ca46632e65a69486557fb73543d57e148fce1653bcc54f5bf69/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/01a3e4e93cde281a25c4601d902180d00a4c3b3d5690773ec0d647ff9076f615/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c1ce0983902c376fb8f09eb7941119e130f8e7330913544c72ed9ab163db4531/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/92e69033e15c2fa060c4de85494afd4be16bcd114140a70ecd330e09b86a16e1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6ca8c25c8dbdaa526233d40076f146642720520e499fa245a581f0c02653d12d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/07c6563a8d05ee83e949095068d2b63064759913edddd41ed1c13ff2468b5d98/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c621d6bfb0115022477192ade84507f1ad808e37400a23c915a6978fa182166a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bb6344119143998fd6f9d69f9d744f265f2fe6f572586bf09fd938ee7a3eeb81/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6c97d089bb22c5baa9b2ebb68790a78a1d34bc7c89949c6982b293f43b8bd5cb/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7d93ff861c230783fb5e0d1be6e35b0c1e8dd118227f35457d8c475889a8c043/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7a920dae338eee5cba3aa4260d0f3c930a2afe29bfa0873256463dc327ae7f14/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/d127e4d60bfaa5946f49a9a078383da7497d0484c57ab45668748dbc8666fc76/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/c42585af12a0d206431e8df555fc860cef8771f9cd2153516f768d2098f1194f/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/acf5b308b203f0468252d4094fd071577497af227484ee7d96c155896984ca2b/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/db5980daa4cccc7897cde8a6853eb8d1fb057037619155fe098de65c222edee1/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/bb0cd8c94a640aa632cdda83438693e6b91530b90319c0f4f90c82a5d085b2c0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/33676355a30efb9844bf2ac13e894398629a847e613ea8f2fae2b575eb9d3866/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/8f4531fe37f1ebad5c7806b4957786e40f2f8904af3ea5ea5772865f8d4f3d9b/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/bd459efd020f78232c1a46304288ca6d5abc8be963b1d65cb39628b888893d3b/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/e6e0f149d1684b27e1f04bdca355f2d4e9bdae524391ff219b0d7f2f2d10e6cf/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/8a860a72a47c62da2fa9a8821964335ae15353c4e1f93bf79c3e08896d9fd4a2/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/7834a4c02f8610082de41ac654cc0f92e50bb16f8dce0a9f9005810a19bfc766/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/e4bdfa1790c2ea00e6f105f7d937437d06127a0db551cdc4ad757a4673f2cb27/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a9e661522c7e03d9f3f8b0acd70c2c4b6464330ff597362d6f64d79c2bce4762/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/b32c78de0ac955680c6672b6289dc5befb18c047c1eae3f8bae2f1ea281ac581/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a901254cf841686bb4ff76cef0b9bc34f1c501df45200a701b91f32bd9f0ea3b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b6cfa8c1cafe39dd2d6be84d124bbe6f919d2b132326189739635c13487e41f2/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/ec811e40f20a99f0cbee593cdd8bc7a9a1aa4ee60a1ddb17cb703eea385eb32c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/22b451b6115f9200a2b3c6582302cdaf2b4e3701d95bfa562c47c17a17373427/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/79f5934bdf298a5e8a7e2faa340fb144b1c71ed1ada0264f308cd1e3d4f8a52f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/de1bcd962a409ce4be10692082e765c9f82bd5a3748cc1de25375ce6360c4350/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1c033f6dc5689464a2a4b6d935c2d5ebaae3c934e17578fb268c816450c3457f/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2809bc9e74d5dd63819b9e3069cefd22bb682fc5e259d712ed836e4a819f73ba/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 31 14:45:59.342918 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.341749    4549 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:59.340901247 +0000 UTC m=+0.067182217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106618 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec246b599922c9270a2278bfee833525 SystemUUID:ec246b59-9922-c927-0a22-78bfee833525 BootID:6ac9ca99-8f81-44c4-9da4-585c73cb9b64 Filesystems:[{Device:/run/containers/storage/overlay-containers/e5ac7137076415b5ebd99ef0e64cbfb1e45ae5c323ab3fcbbfcd8841553559e0/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/309034fe-5db8-476f-b335-41a37a2c9464/volumes/kubernetes.io~projected/kube-api-access-7hpkn DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32080cb0-02b5-44e5-a7b2-3f33c5f97f48/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ebed5d09-55bf-4935-8ea9-55360332d683/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8120bc72-1aec-417f-9be2-4bc93ef332b5/volumes/kubernetes.io~projected/kube-api-access-kqpnm DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aecc9d82-e0fe-40a1-9bf2-c924e81b4433/volumes/kubernetes.io~projected/kube-api-access-m9jwm DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a28abdca-0d74-42d8-b328-520e11f32600/volumes/kubernetes.io~projected/kube-api-access-2w6lm DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eacc6dd145f1a13bdf6553cc6ad3b4859f1e7d6193252bfbeceeb7c03dc2cfb3/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volumes/kubernetes.io~projected/kube-api-access-28t9l DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7197ee7eaf92734a525d115c8b9f76cf2ce84f6a379f7d7136252f2faaf79ad3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55deca85afa0776b59916009468ab2b1f55e119417667c84c943632b965c2888/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c887f94b8b5bda23a70e00a47930a2903c5f51d4ffded2cb8b0ab8ae0cfdd37/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a51637a9402498c4a46e44f9fd8e5efd3c2d38317c11f355534ba6994b12e1d/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f63e0003-a810-483a-8ab0-4e01ea21f4e9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7dc686b7-618e-47ee-b28e-35f1cbee4212/volumes/kubernetes.io~projected/kube-api-access-fmb6c DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3449570bd50fc1077221c166489e0ff0986bfa551fab01c66397d43cb9bd335e/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/97e554f5-645d-42c9-b73d-8c1f8fa88d80/volumes/kubernetes.io~projected/kube-api-access-qm2c8 DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a32e72b8-08d0-4824-9b97-305921ef0026/volumes/kubernetes.io~projected/kube-api-access-zf6tx DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1bbabbcfb0fc046ff427868183ab054b3b663bb434f147b8a4c4265b014c5df/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a902571-08a4-48f8-9384-caed28d8893e/volumes/kubernetes.io~projected/kube-api-access-zvbs7 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5422f2a0dc6afba272ba0fe742f4b3ddd1a2b18d5135d7059a6a6a88e5e4b0b/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97f80799e649c811257a2ae1ea2ba67bd358ed4f21b3d39711475e13aba32d6b/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/381c174e-6f38-4afb-a8f1-08514439fa50/volumes/kubernetes.io~projected/kube-api-access-62kcv DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d04e5de2-0045-40dc-b5cf-99533e3724c2/volumes/kubernetes.io~projected/kube-api-access-zddg5 DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d77bf10-10c6-445e-9c93-9832cc17ee67/volumes/kubernetes.io~projected/kube-api-access-sprcb DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4c37336-0737-4801-9213-08d878a4e6ad/volumes/kubernetes.io~projected/kube-api-access-84zqg DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/747fad2563ade1c6a8812d1bb09dd7731066d7fbdab9ef350ee2b638506fa907/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bd6a76-b8c7-45e9-8d16-678fd598eecb/volumes/kubernetes.io~projected/kube-api-access-wjfhd DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1c844afc-670c-45f5-82bb-9f53eb55ac91/volumes/kubernetes.io~projected/kube-api-access-cjmdb DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f63ce461622d7d6b5a70b3dfadcd6e5c8eb8f98bc0c8f922b5f9d55f1276e063/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb4f70dd5023dbb6230463c444c208908906ab4a8bdd491bae43cdd4949b5582/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e42ce53e-cac9-470c-aa88-cd889695d7c3/volumes/kubernetes.io~projected/kube-api-access-6h755 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0c887f94b8b5bda MacAddress:4a:d5:58:8f:bf:c9 Speed:10000 Mtu:8901} {Name:3449570bd50fc10 MacAddress:5e:b2:fa:43:70:ce Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:71:ac:00:91 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:2e:18:15:52:c4 Speed:0 Mtu:8901} {Name:c781b4039b9c4a2 MacAddress:36:57:77:49:8c:67 Speed:10000 Mtu:8901} {Name:e5ac7137076415b MacAddress:62:66:cc:80:83:53 Speed:10000 Mtu:8901} {Name:eacc6dd145f1a13 MacAddress:46:b2:b6:cd:45:92 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:71:ac:00:91 Speed:0 Mtu:9001} {Name:f63ce461622d7d6 MacAddress:16:ed:f7:0b:79:b8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c2:11:87:9c:08:02 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:19:bb:7f:83:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 14:45:59.343130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.342925    4549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 14:45:59.343130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.342999    4549 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 14:45:59.343366 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343338    4549 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 14:45:59.343522 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343368    4549 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-115.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 14:45:59.343666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343527    4549 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 14:45:59.343666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343547    4549 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 14:45:59.343666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343588    4549 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 14:45:59.343666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343631    4549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 14:45:59.343830 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343773    4549 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 14:45:59.344006 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.343991    4549 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 14:45:59.344138 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344127    4549 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 14:45:59.344198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344185    4549 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 14:45:59.344245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344199    4549 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 14:45:59.344245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344215    4549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 14:45:59.344245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344230    4549 kubelet.go:405] "Adding apiserver pod source"
Oct 31 14:45:59.344245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.344241    4549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 14:45:59.345615 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.345308    4549 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 14:45:59.345615 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.345324    4549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 14:45:59.345993 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.345965    4549 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 14:45:59.348696 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.348679    4549 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 14:45:59.348696 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.348695    4549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 14:45:59.348828 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.348813    4549 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 14:45:59.349092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349075    4549 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 14:45:59.349172 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349124    4549 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 14:45:59.349248 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349237    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 14:45:59.349302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349258    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 14:45:59.349302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349268    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 14:45:59.349302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349277    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 14:45:59.349302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349286    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 14:45:59.349302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349294    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349306    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349314    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349325    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349333    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349351    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349364    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349399    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 14:45:59.349525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.349407    4549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 14:45:59.355740 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.355722    4549 server.go:1263] "Started kubelet"
Oct 31 14:45:59.356540 ip-10-0-139-115 systemd[1]: Started Kubernetes Kubelet.
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.357142    4549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.357304    4549 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.359873    4549 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.359929    4549 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.361087    4549 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.363749    4549 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 14:45:59.364942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.364183    4549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.366727    4549 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.366740    4549 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.366875    4549 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.367156    4549 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.367533    4549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:20 +0000 UTC" deadline="2028-07-30 18:22:05.050681088 +0000 UTC"
Oct 31 14:45:59.367676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.367606    4549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15315h36m5.6830796s"
Oct 31 14:45:59.368293 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368262    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="309034fe-5db8-476f-b335-41a37a2c9464" volumeName="kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368295    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8120bc72-1aec-417f-9be2-4bc93ef332b5" volumeName="kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368305    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" volumeName="kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368312    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e42ce53e-cac9-470c-aa88-cd889695d7c3" volumeName="kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368320    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63e0003-a810-483a-8ab0-4e01ea21f4e9" volumeName="kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368327    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="381c174e-6f38-4afb-a8f1-08514439fa50" volumeName="kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368334    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32e72b8-08d0-4824-9b97-305921ef0026" volumeName="kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy" seLinuxMountContext=""
Oct 31 14:45:59.368349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368346    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d04e5de2-0045-40dc-b5cf-99533e3724c2" volumeName="kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368355    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32e72b8-08d0-4824-9b97-305921ef0026" volumeName="kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368362    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32e72b8-08d0-4824-9b97-305921ef0026" volumeName="kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368370    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a32e72b8-08d0-4824-9b97-305921ef0026" volumeName="kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368377    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aecc9d82-e0fe-40a1-9bf2-c924e81b4433" volumeName="kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368383    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8120bc72-1aec-417f-9be2-4bc93ef332b5" volumeName="kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368390    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c844afc-670c-45f5-82bb-9f53eb55ac91" volumeName="kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368398    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="381c174e-6f38-4afb-a8f1-08514439fa50" volumeName="kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368405    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d77bf10-10c6-445e-9c93-9832cc17ee67" volumeName="kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368429    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a902571-08a4-48f8-9384-caed28d8893e" volumeName="kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368435    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aecc9d82-e0fe-40a1-9bf2-c924e81b4433" volumeName="kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368443    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e42ce53e-cac9-470c-aa88-cd889695d7c3" volumeName="kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368450    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4c37336-0737-4801-9213-08d878a4e6ad" volumeName="kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368458    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" volumeName="kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368464    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" volumeName="kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368471    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d04e5de2-0045-40dc-b5cf-99533e3724c2" volumeName="kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368477    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32080cb0-02b5-44e5-a7b2-3f33c5f97f48" volumeName="kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368484    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc686b7-618e-47ee-b28e-35f1cbee4212" volumeName="kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368490    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63e0003-a810-483a-8ab0-4e01ea21f4e9" volumeName="kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368497    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aecc9d82-e0fe-40a1-9bf2-c924e81b4433" volumeName="kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368502    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="381c174e-6f38-4afb-a8f1-08514439fa50" volumeName="kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368512    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d77bf10-10c6-445e-9c93-9832cc17ee67" volumeName="kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368518    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d77bf10-10c6-445e-9c93-9832cc17ee67" volumeName="kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368525    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a902571-08a4-48f8-9384-caed28d8893e" volumeName="kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368536    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dc686b7-618e-47ee-b28e-35f1cbee4212" volumeName="kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368542    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e554f5-645d-42c9-b73d-8c1f8fa88d80" volumeName="kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368547    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e554f5-645d-42c9-b73d-8c1f8fa88d80" volumeName="kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368556    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a28abdca-0d74-42d8-b328-520e11f32600" volumeName="kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368562    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" volumeName="kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368586    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e42ce53e-cac9-470c-aa88-cd889695d7c3" volumeName="kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368597    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebed5d09-55bf-4935-8ea9-55360332d683" volumeName="kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368606    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebed5d09-55bf-4935-8ea9-55360332d683" volumeName="kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368616    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63e0003-a810-483a-8ab0-4e01ea21f4e9" volumeName="kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368627    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63e0003-a810-483a-8ab0-4e01ea21f4e9" volumeName="kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368636    4549 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f63e0003-a810-483a-8ab0-4e01ea21f4e9" volumeName="kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368646    4549 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 14:45:59.368674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.368652    4549 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.381556    4549 factory.go:55] Registering systemd factory
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.381601    4549 factory.go:223] Registration of the systemd container factory successfully
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382000    4549 factory.go:153] Registering CRI-O factory
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382024    4549 factory.go:223] Registration of the crio container factory successfully
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382076    4549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382094    4549 factory.go:103] Registering Raw factory
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382110    4549 manager.go:1199] Started watching for new ooms in manager
Oct 31 14:45:59.385556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.382439    4549 manager.go:319] Starting recovery of all containers
Oct 31 14:45:59.409339 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.409294    4549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 14:45:59.411364 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.411343    4549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 14:45:59.411364 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.411365    4549 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 14:45:59.411497 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.411411    4549 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 14:45:59.411497 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.411448    4549 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 14:45:59.448248 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448173    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="8285a082840549c2d4fca7ff6ba60ded0b9cd9f8d1e56b45daa4a4e1106c4644" exitCode=0
Oct 31 14:45:59.448396 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448383    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="61cb5e680eae8790504760658411cc01096d4c5bfa453c756abfb965bcf44411" exitCode=0
Oct 31 14:45:59.448460 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448450    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="4ea71fc251db50af247365a56f1820d7cf8d8d66286f6d6906b911f5d74d19f0" exitCode=0
Oct 31 14:45:59.448528 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448518    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="fbb6c644ba67676fbf7917a515b878989b34bb1362755de9fafe5a1b2de2fc12" exitCode=0
Oct 31 14:45:59.448598 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448588    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="4d4109138569a23c5e91310cd68856ab162f5cfd22f241a12d64580d9fceb672" exitCode=0
Oct 31 14:45:59.448659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.448649    4549 generic.go:356] "Generic (PLEG): container finished" podID="a32e72b8-08d0-4824-9b97-305921ef0026" containerID="9d9426afe682cb02d4bbb029e8b9dcd90860495be6336d3b2c50c4e4e00d3f30" exitCode=0
Oct 31 14:45:59.466410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.466377    4549 generic.go:356] "Generic (PLEG): container finished" podID="3b1c519a9e746dec2e3a5ff69d67b2ad" containerID="9685e69bebc5487c0a72b5d576cfd25d61904fd29fd777e01b3899eb6f3c2ab9" exitCode=0
Oct 31 14:45:59.467665 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.467643    4549 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found"
Oct 31 14:45:59.469598 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.469559    4549 manager.go:324] Recovery completed
Oct 31 14:45:59.483732 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.483707    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.485898 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.485874    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.486020 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.485915    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.486020 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.485931    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.487914 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.487896    4549 cpu_manager.go:226] "Starting" policy="none"
Oct 31 14:45:59.487914 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.487910    4549 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 14:45:59.488028 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.487926    4549 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 14:45:59.488028 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.487901    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.488129 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488086    4549 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 14:45:59.488129 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488102    4549 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 14:45:59.488129 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488125    4549 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 14:45:59.488219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488133    4549 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 14:45:59.488219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488141    4549 policy_none.go:50] "Start"
Oct 31 14:45:59.488219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488156    4549 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 14:45:59.488219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488166    4549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 14:45:59.488333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488296    4549 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 14:45:59.488333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488303    4549 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 14:45:59.488333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488309    4549 policy_none.go:44] "Start"
Oct 31 14:45:59.488879 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488863    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.488945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488896    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.488945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.488915    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.492722 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.492695    4549 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 14:45:59.495305 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495293    4549 manager.go:341] "Starting Device Plugin manager"
Oct 31 14:45:59.495384 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.495315    4549 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 14:45:59.495384 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495328    4549 server.go:87] "Starting device plugin registration server"
Oct 31 14:45:59.495631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495616    4549 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 14:45:59.495680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495630    4549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 14:45:59.495757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495738    4549 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 14:45:59.495879 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495846    4549 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 14:45:59.495944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495888    4549 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 14:45:59.495944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495895    4549 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 14:45:59.496028 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.495947    4549 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:59.495877327Z","UUID":"a96056a2-7eb4-4376-aa5d-3f3db5befc71","Handler":null,"Name":"","Endpoint":""}
Oct 31 14:45:59.497326 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.497310    4549 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 14:45:59.497416 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.497348    4549 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-115.us-west-1.compute.internal\" not found"
Oct 31 14:45:59.497416 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.497398    4549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 14:45:59.497502 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.497419    4549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 14:45:59.511637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.511600    4549 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"]
Oct 31 14:45:59.511757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.511723    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.512524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.512508    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.512620 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.512540    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.512620 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.512552    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.512815 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.512803    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.512990 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.512975    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.513556 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513539    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.513638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513586    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.513638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513601    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.513638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513634    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.513732 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513655    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.513732 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513664    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.513817 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.513805    4549 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.513910 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513877    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerStarted","Data":"5837b78d01ace63d2c90cc8ec5b5df40047c7d6b8b18fbdabf461aa0d9753560"}
Oct 31 14:45:59.513950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513925    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.513980 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.513925    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerDied","Data":"9685e69bebc5487c0a72b5d576cfd25d61904fd29fd777e01b3899eb6f3c2ab9"}
Oct 31 14:45:59.514026 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514008    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"3b1c519a9e746dec2e3a5ff69d67b2ad","Type":"ContainerStarted","Data":"085d599335eeb9b124d5a1bf7dc1f59088c518e6cc4ad34d99458059363732bb"}
Oct 31 14:45:59.514064 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514035    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"d57afef5546aa1919f861928dcb50ce6","Type":"ContainerStarted","Data":"a371bd503cce52b280f746794adc865dcc083203e9ddb201e6e8bdb515a63f5d"}
Oct 31 14:45:59.514105 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514069    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal" event={"ID":"d57afef5546aa1919f861928dcb50ce6","Type":"ContainerStarted","Data":"45c023e54fb199bc1b27ebfb1a71587798d92ef47b1d7f564fb018cc0b192515"}
Oct 31 14:45:59.514654 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514639    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.514723 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514669    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.514723 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.514683    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.514842 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.514829    4549 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-115.us-west-1.compute.internal\" not found" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.561757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.561722    4549 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:59.561757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.561726    4549 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:59.562001 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.561987    4549 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 14:45:59.596087 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.596058    4549 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 14:45:59.597156 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.597141    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 14:45:59.597229 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.597171    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 14:45:59.597229 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.597185    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 14:45:59.597229 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.597223    4549 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.610092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.610069    4549 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:59.611894 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.611877    4549 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 14:45:59.662794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.662755    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.662956 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.662851    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.662956 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.662884    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.668338 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.668320    4549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.711469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.711390    4549 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.711469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.711461    4549 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.712069 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.712055    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeNotReady"
Oct 31 14:45:59.712124 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.712085    4549 setters.go:546] "Node became not ready" node="ip-10-0-139-115.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T14:45:59Z","lastTransitionTime":"2026-10-31T14:45:59Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 14:45:59.722594 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.722558    4549 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-115.us-west-1.compute.internal" is owned by different node (oldNodeID="47c790e4-f396-4ba4-9bcf-a4f337ac75cc", newNodeID=""), cleaning up...
Oct 31 14:45:59.726554 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.726535    4549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:45:59.726668 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.726598    4549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.726668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.726616    4549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763497    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763607    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d57afef5546aa1919f861928dcb50ce6-config\") pod \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"d57afef5546aa1919f861928dcb50ce6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763641    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763610    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763710    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.763786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.763757    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b1c519a9e746dec2e3a5ff69d67b2ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" (UID: \"3b1c519a9e746dec2e3a5ff69d67b2ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:45:59.783833 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:45:59.783811    4549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:45:59.783965 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:45:59.783853    4549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:46:00.345359 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.345324    4549 apiserver.go:52] "Watching apiserver"
Oct 31 14:46:00.470656 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.470628    4549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:46:00.470801 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.470787    4549 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:46:00.498165 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.498141    4549 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 14:46:00.498636 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.498615    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-s2smk","openshift-dns/node-resolver-bhqss","kube-system/konnectivity-agent-ktm59","kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal","openshift-insights/insights-runtime-extractor-jk8rn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp","openshift-multus/multus-additional-cni-plugins-lnk8v","openshift-ovn-kubernetes/ovnkube-node-x4sfm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-26fwk","openshift-multus/network-metrics-daemon-64bfw","openshift-dns/dns-default-ls8dq","openshift-ingress-canary/ingress-canary-cblh8","kube-system/global-pull-secret-syncer-hmc4m","openshift-image-registry/node-ca-gjwxr","openshift-insights/periodic-gathering-k4hdw-tw97l","openshift-network-diagnostics/network-check-target-ljbk9","openshift-network-operator/iptables-alerter-ccjhc","openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8","openshift-multus/multus-swww9"]
Oct 31 14:46:00.499074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.499058    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:00.503361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.503341    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.506818 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.506795    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.531031 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.531007    4549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:46:00.531167 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.531055    4549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:46:00.531486 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.531474    4549 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 14:46:00.531676 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.531507    4549 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-115.us-west-1.compute.internal"
Oct 31 14:46:00.557653 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557621    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 14:46:00.557798 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557752    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.557897 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557883    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-whbtk\""
Oct 31 14:46:00.557942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557896    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.557942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557907    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\""
Oct 31 14:46:00.558018 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557956    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 14:46:00.558018 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557969    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 14:46:00.558018 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557931    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\""
Oct 31 14:46:00.558018 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.557989    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 14:46:00.558139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558047    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.558139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558085    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\""
Oct 31 14:46:00.558139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558080    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558300 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558283    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558300 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558274    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558293    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558311    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558299    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558314    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558376    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.558417 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558391    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 14:46:00.558657 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558629    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.558657 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558639    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558706    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 14:46:00.558793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558737    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\""
Oct 31 14:46:00.558793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558750    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 14:46:00.558793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558743    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.558793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558765    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558797    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558812    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558817    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558827    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558827    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558866    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558944    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.558978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558956    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.558995    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559019    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559127    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559130    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559219    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559249    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 14:46:00.559308 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.559269    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\""
Oct 31 14:46:00.568394 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.568375    4549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 14:46:00.580750 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.580731    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\""
Oct 31 14:46:00.603086 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.603025    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\""
Oct 31 14:46:00.603179 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.603095    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 14:46:00.603179 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.603104    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\""
Oct 31 14:46:00.603179 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.603114    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\""
Oct 31 14:46:00.605178 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.605162    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\""
Oct 31 14:46:00.607630 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.607617    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 14:46:00.608557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608533    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.608671 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608617    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\""
Oct 31 14:46:00.608735 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608681    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 14:46:00.608814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608735    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\""
Oct 31 14:46:00.608814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608740    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.608814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608791    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 14:46:00.608814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608796    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608818    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608867    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608870    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608910    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j8ksf\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608887    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608933    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 14:46:00.608987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608947    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.609273 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608996    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 14:46:00.609273 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.608969    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\""
Oct 31 14:46:00.609593 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.609558    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 14:46:00.667655 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.667618    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zqg\" (UniqueName: \"kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.667655 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.667655    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.667867 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.667673    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvbs7\" (UniqueName: \"kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:46:00.667991 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.667904    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.667991 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.667953    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.668080 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668012    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.668080 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668033    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.668171 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668073    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668171 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668100    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.668171 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668101    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.668171 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668163    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668187    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668214    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668239    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddg5\" (UniqueName: \"kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668241    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42ce53e-cac9-470c-aa88-cd889695d7c3-tmp-dir\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668275    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.668358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668342    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04e5de2-0045-40dc-b5cf-99533e3724c2-data-volume\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.668637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668371    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668391    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:46:00.668637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668406    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.668637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668459    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.668637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668615    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668649    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-config\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668666    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668684    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668739    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668763    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668779    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668788    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ebed5d09-55bf-4935-8ea9-55360332d683-konnectivity-ca\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:46:00.668884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668824    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668929    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668947    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668977    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668990    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-tmp\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669000    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.668988    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e42ce53e-cac9-470c-aa88-cd889695d7c3-config-volume\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669025    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669064    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669076    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dc686b7-618e-47ee-b28e-35f1cbee4212-tmp-dir\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.669227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669136    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669240    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669314    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669340    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a902571-08a4-48f8-9384-caed28d8893e-cert\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669344    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669432    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.669680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669448    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/381c174e-6f38-4afb-a8f1-08514439fa50-etc-tuned\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669795    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/97e554f5-645d-42c9-b73d-8c1f8fa88d80-serviceca\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669388    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669843    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669868    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669882    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669898    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmb6c\" (UniqueName: \"kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669913    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669930    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669953    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669970    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.669988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669984    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.669998    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670013    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670026    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2c8\" (UniqueName: \"kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670058    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670078    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprcb\" (UniqueName: \"kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670099    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670113    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670132    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670147    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670161    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670180    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670196    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjmdb\" (UniqueName: \"kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb\") pod \"migrator-66df5d496b-t42s8\" (UID: \"1c844afc-670c-45f5-82bb-9f53eb55ac91\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670210    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670225    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670240    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670254    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670267    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670282    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670304    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670318    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670338    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670353    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670367    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670383    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6tx\" (UniqueName: \"kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670400    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670415    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpnm\" (UniqueName: \"kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670432    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kcv\" (UniqueName: \"kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670449    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.670461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670470    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670485    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670498    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670513    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670529    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670542    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670555    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670588    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670603    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670616    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670631    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670650    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670665    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670680    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670694    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h755\" (UniqueName: \"kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670710    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670724    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670740    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670755    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jwm\" (UniqueName: \"kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670772    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670788    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28t9l\" (UniqueName: \"kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670803    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670818    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670833    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670849    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpkn\" (UniqueName: \"kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670865    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.670886    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.671065    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-key\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.671642 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.671471    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-original-pull-secret\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.671876    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.671999    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ebed5d09-55bf-4935-8ea9-55360332d683-agent-certs\") pod \"konnectivity-agent-ktm59\" (UID: \"ebed5d09-55bf-4935-8ea9-55360332d683\") " pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672014    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cni-binary-copy\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672086    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672136    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672202    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32e72b8-08d0-4824-9b97-305921ef0026-cni-binary-copy\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672363    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-ovnkube-script-lib\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672489    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-daemon-config\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672546    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8120bc72-1aec-417f-9be2-4bc93ef332b5-iptables-alerter-script\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672592    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3d77bf10-10c6-445e-9c93-9832cc17ee67-signing-cabundle\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.672866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.672675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f63e0003-a810-483a-8ab0-4e01ea21f4e9-env-overrides\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.723415 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.723313    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zqg\" (UniqueName: \"kubernetes.io/projected/f4c37336-0737-4801-9213-08d878a4e6ad-kube-api-access-84zqg\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.723415 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.723386    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddg5\" (UniqueName: \"kubernetes.io/projected/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-api-access-zddg5\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.723733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.723559    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvbs7\" (UniqueName: \"kubernetes.io/projected/5a902571-08a4-48f8-9384-caed28d8893e-kube-api-access-zvbs7\") pod \"ingress-canary-cblh8\" (UID: \"5a902571-08a4-48f8-9384-caed28d8893e\") " pod="openshift-ingress-canary/ingress-canary-cblh8"
Oct 31 14:46:00.723814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.723726    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6lm\" (UniqueName: \"kubernetes.io/projected/a28abdca-0d74-42d8-b328-520e11f32600-kube-api-access-2w6lm\") pod \"network-check-target-ljbk9\" (UID: \"a28abdca-0d74-42d8-b328-520e11f32600\") " pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:46:00.726738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.726715    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmb6c\" (UniqueName: \"kubernetes.io/projected/7dc686b7-618e-47ee-b28e-35f1cbee4212-kube-api-access-fmb6c\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.726933 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.726913    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpnm\" (UniqueName: \"kubernetes.io/projected/8120bc72-1aec-417f-9be2-4bc93ef332b5-kube-api-access-kqpnm\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.727114 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.727099    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6tx\" (UniqueName: \"kubernetes.io/projected/a32e72b8-08d0-4824-9b97-305921ef0026-kube-api-access-zf6tx\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.728254 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.728233    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2c8\" (UniqueName: \"kubernetes.io/projected/97e554f5-645d-42c9-b73d-8c1f8fa88d80-kube-api-access-qm2c8\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.728353 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.728337    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kcv\" (UniqueName: \"kubernetes.io/projected/381c174e-6f38-4afb-a8f1-08514439fa50-kube-api-access-62kcv\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.728728 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.728710    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjmdb\" (UniqueName: \"kubernetes.io/projected/1c844afc-670c-45f5-82bb-9f53eb55ac91-kube-api-access-cjmdb\") pod \"migrator-66df5d496b-t42s8\" (UID: \"1c844afc-670c-45f5-82bb-9f53eb55ac91\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-t42s8"
Oct 31 14:46:00.728808 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.728764    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"periodic-gathering-k4hdw-tw97l\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") " pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:46:00.728808 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.728787    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpkn\" (UniqueName: \"kubernetes.io/projected/309034fe-5db8-476f-b335-41a37a2c9464-kube-api-access-7hpkn\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:00.729090 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.729072    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28t9l\" (UniqueName: \"kubernetes.io/projected/f63e0003-a810-483a-8ab0-4e01ea21f4e9-kube-api-access-28t9l\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.729608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.729590    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprcb\" (UniqueName: \"kubernetes.io/projected/3d77bf10-10c6-445e-9c93-9832cc17ee67-kube-api-access-sprcb\") pod \"service-ca-5fffc97bdf-26fwk\" (UID: \"3d77bf10-10c6-445e-9c93-9832cc17ee67\") " pod="openshift-service-ca/service-ca-5fffc97bdf-26fwk"
Oct 31 14:46:00.729772 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.729755    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h755\" (UniqueName: \"kubernetes.io/projected/e42ce53e-cac9-470c-aa88-cd889695d7c3-kube-api-access-6h755\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.730253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.730238    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jwm\" (UniqueName: \"kubernetes.io/projected/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-kube-api-access-m9jwm\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.771516 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771482    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.771516 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771517    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771544    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771565    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771624    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771652    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-sys\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771666    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771670    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d04e5de2-0045-40dc-b5cf-99533e3724c2-crio-socket\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771659    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-bin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771630    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97e554f5-645d-42c9-b73d-8c1f8fa88d80-host\") pod \"node-ca-gjwxr\" (UID: \"97e554f5-645d-42c9-b73d-8c1f8fa88d80\") " pod="openshift-image-registry/node-ca-gjwxr"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771630    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-cnibin\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771722    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-netns\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.771755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771743    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771761    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771786    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771806    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-systemd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771812    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771837    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771843    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-etc-kubernetes\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771848    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-cnibin\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771862    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771884    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-var-lib-kubelet\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771887    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-etc-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771916    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771926    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-kubelet-config\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.771945    4549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771952    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771971    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.771987    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772005    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772024    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772053    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-socket-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772064    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772034    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-ovn\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772089    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-k8s-cni-cncf-io\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.772107    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.272080996 +0000 UTC m=+1.998361977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772122    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-system-cni-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772133    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-kubernetes\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772130    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-socket-dir-parent\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772157    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32e72b8-08d0-4824-9b97-305921ef0026-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lnk8v\" (UID: \"a32e72b8-08d0-4824-9b97-305921ef0026\") " pod="openshift-multus/multus-additional-cni-plugins-lnk8v"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772164    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772166    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.772184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772199    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-hostroot\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772205    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772228    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772242    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772258    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772267    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-slash\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772269    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-log-socket\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772275    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772306    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8120bc72-1aec-417f-9be2-4bc93ef332b5-host-slash\") pod \"iptables-alerter-ccjhc\" (UID: \"8120bc72-1aec-417f-9be2-4bc93ef332b5\") " pod="openshift-network-operator/iptables-alerter-ccjhc"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772311    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-system-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772307    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772337    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-multus-certs\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.772353    4549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772392    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.772408    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.272393434 +0000 UTC m=+1.998674394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : secret "metrics-daemon-secret" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772349    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-run\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772433    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772456    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772472    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772485    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772498    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772519    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772535    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-sys-fs\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772541    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772556    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7dc686b7-618e-47ee-b28e-35f1cbee4212-hosts-file\") pod \"node-resolver-bhqss\" (UID: \"7dc686b7-618e-47ee-b28e-35f1cbee4212\") " pod="openshift-dns/node-resolver-bhqss"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772564    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772581    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772609    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-cni-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772616    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-run-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772626    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-host\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772633    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-kubelet\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772636    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-systemd-units\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772660    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772697    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772700    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772736    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772761    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-modprobe-d\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772761    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772781    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772792    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-run-netns\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772798    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772804    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-registration-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772825    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-multus-conf-dir\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772829    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772859    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772894    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-device-dir\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772895    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-lib-modules\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772922    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772957    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-node-log\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.772986    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773018    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f4c37336-0737-4801-9213-08d878a4e6ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-7cjzp\" (UID: \"f4c37336-0737-4801-9213-08d878a4e6ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7cjzp"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773064    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32080cb0-02b5-44e5-a7b2-3f33c5f97f48-dbus\") pod \"global-pull-secret-syncer-hmc4m\" (UID: \"32080cb0-02b5-44e5-a7b2-3f33c5f97f48\") " pod="kube-system/global-pull-secret-syncer-hmc4m"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773067    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-cni-multus\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773090    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773121    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773145    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773169    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-var-lib-openvswitch\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773175    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773199    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773210    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysctl-conf\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.773219    4549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773222    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773245    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773249    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aecc9d82-e0fe-40a1-9bf2-c924e81b4433-host-var-lib-kubelet\") pod \"multus-swww9\" (UID: \"aecc9d82-e0fe-40a1-9bf2-c924e81b4433\") " pod="openshift-multus/multus-swww9"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.773255    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.273242667 +0000 UTC m=+1.999523625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.773255    4549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773279    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-netd\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773286    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-cni-bin\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:00.773318    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.273307871 +0000 UTC m=+1.999588828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773333    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773368    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773383    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773394    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f63e0003-a810-483a-8ab0-4e01ea21f4e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4sfm\" (UID: \"f63e0003-a810-483a-8ab0-4e01ea21f4e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773428    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-systemd\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.773395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.773446    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/381c174e-6f38-4afb-a8f1-08514439fa50-etc-sysconfig\") pod \"tuned-s2smk\" (UID: \"381c174e-6f38-4afb-a8f1-08514439fa50\") " pod="openshift-cluster-node-tuning-operator/tuned-s2smk"
Oct 31 14:46:00.908626 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:00.908538    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-t42s8_1c844afc-670c-45f5-82bb-9f53eb55ac91/migrator/0.log"
Oct 31 14:46:01.277901 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:01.277866    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:01.278070 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:01.277925    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:01.278070 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278011    4549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 14:46:01.278070 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278016    4549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:46:01.278245 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278079    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:02.278053454 +0000 UTC m=+3.004334409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:46:01.278245 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278096    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:02.278087478 +0000 UTC m=+3.004368433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : secret "metrics-daemon-secret" not found
Oct 31 14:46:01.278245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:01.278113    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:01.278245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:01.278162    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:01.278245 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278227    4549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:46:01.278445 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278253    4549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:01.278445 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278261    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:02.278250353 +0000 UTC m=+3.004531309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:46:01.278445 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:01.278337    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:02.278324118 +0000 UTC m=+3.004605106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:01.992466 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:01.992436    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:46:02.285444 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:02.285412    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:02.285463    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:02.285504    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:02.285538    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285595    4549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285633    4549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285653    4549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285657    4549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 14:46:02.285684 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285682    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls podName:e42ce53e-cac9-470c-aa88-cd889695d7c3 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:04.285657859 +0000 UTC m=+5.011938816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls") pod "dns-default-ls8dq" (UID: "e42ce53e-cac9-470c-aa88-cd889695d7c3") : secret "dns-default-metrics-tls" not found
Oct 31 14:46:02.285943 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285699    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:04.285688593 +0000 UTC m=+5.011969549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:02.285943 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285714    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs podName:309034fe-5db8-476f-b335-41a37a2c9464 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:04.28570704 +0000 UTC m=+5.011987997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs") pod "network-metrics-daemon-64bfw" (UID: "309034fe-5db8-476f-b335-41a37a2c9464") : secret "metrics-daemon-secret" not found
Oct 31 14:46:02.285943 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:02.285727    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:04.285719715 +0000 UTC m=+5.012000671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:46:04.301664 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.301623    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.301677    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.301778    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.301851    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:04.301898    4549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:04.301971    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:08.301951803 +0000 UTC m=+9.028232760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : configmap "kube-rbac-proxy" not found
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:04.301969    4549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.302002    4549 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 14:46:04.302243 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:04.302024    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:08.302009183 +0000 UTC m=+9.028290142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:04.304154 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.304137    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309034fe-5db8-476f-b335-41a37a2c9464-metrics-certs\") pod \"network-metrics-daemon-64bfw\" (UID: \"309034fe-5db8-476f-b335-41a37a2c9464\") " pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:04.304313 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.304296    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e42ce53e-cac9-470c-aa88-cd889695d7c3-metrics-tls\") pod \"dns-default-ls8dq\" (UID: \"e42ce53e-cac9-470c-aa88-cd889695d7c3\") " pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:04.400437 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.400406    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-64bfw"
Oct 31 14:46:04.403659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.403639    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:04.807634 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.807601    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-64bfw"]
Oct 31 14:46:04.827028 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:04.826992    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309034fe_5db8_476f_b335_41a37a2c9464.slice/crio-8428903fefb49fa10ba2e26350a17066faeca57a03c9abb5773b03c899d72806 WatchSource:0}: Error finding container 8428903fefb49fa10ba2e26350a17066faeca57a03c9abb5773b03c899d72806: Status 404 returned error can't find the container with id 8428903fefb49fa10ba2e26350a17066faeca57a03c9abb5773b03c899d72806
Oct 31 14:46:04.828973 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.828960    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:46:04.858084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:04.858056    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ls8dq"]
Oct 31 14:46:04.861271 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:04.861249    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42ce53e_cac9_470c_aa88_cd889695d7c3.slice/crio-40581595e1b3ab2bafb04740b3e103f6945cdd1240f080e0de65d801f4294d58 WatchSource:0}: Error finding container 40581595e1b3ab2bafb04740b3e103f6945cdd1240f080e0de65d801f4294d58: Status 404 returned error can't find the container with id 40581595e1b3ab2bafb04740b3e103f6945cdd1240f080e0de65d801f4294d58
Oct 31 14:46:05.487439 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:05.487409    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ls8dq" event={"ID":"e42ce53e-cac9-470c-aa88-cd889695d7c3","Type":"ContainerStarted","Data":"40581595e1b3ab2bafb04740b3e103f6945cdd1240f080e0de65d801f4294d58"}
Oct 31 14:46:05.488513 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:05.488489    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64bfw" event={"ID":"309034fe-5db8-476f-b335-41a37a2c9464","Type":"ContainerStarted","Data":"8428903fefb49fa10ba2e26350a17066faeca57a03c9abb5773b03c899d72806"}
Oct 31 14:46:06.494917 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:06.494886    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64bfw" event={"ID":"309034fe-5db8-476f-b335-41a37a2c9464","Type":"ContainerStarted","Data":"17c211f0bd2290b9548bd89c450f9bfc9c4fa88d048adcf73eddc9f0b859e63d"}
Oct 31 14:46:06.495319 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:06.494948    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-64bfw" event={"ID":"309034fe-5db8-476f-b335-41a37a2c9464","Type":"ContainerStarted","Data":"b4ae35a71f022a2bf425491abae1187c19bcad8bf1a3e85ebdf92911feb0aaf5"}
Oct 31 14:46:07.344501 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:07.344476    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:07.502561 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:07.502526    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ls8dq" event={"ID":"e42ce53e-cac9-470c-aa88-cd889695d7c3","Type":"ContainerStarted","Data":"ce32b9ff44ab269925cb686f9c3b886493ec6c730cf05ea6731d436ffeede7db"}
Oct 31 14:46:07.502956 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:07.502626    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ls8dq" event={"ID":"e42ce53e-cac9-470c-aa88-cd889695d7c3","Type":"ContainerStarted","Data":"b6c92c5d220d2c4399b1406329536e8b6f09c696c313e91cb6daabb75cfaab28"}
Oct 31 14:46:07.502956 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:07.502759    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:08.329997 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:08.329961    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:08.330184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:08.330009    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:08.330184 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:08.330117    4549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:08.330184 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:08.330181    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls podName:d04e5de2-0045-40dc-b5cf-99533e3724c2 nodeName:}" failed. No retries permitted until 2026-10-31 14:46:16.330168426 +0000 UTC m=+17.056449382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk8rn" (UID: "d04e5de2-0045-40dc-b5cf-99533e3724c2") : secret "insights-runtime-extractor-tls" not found
Oct 31 14:46:08.330493 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:08.330475    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d04e5de2-0045-40dc-b5cf-99533e3724c2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:08.518906 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:08.518878    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:08.519269 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:08.519127    4549 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 14:46:09.875458 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:09.875428    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeReady"
Oct 31 14:46:16.384701 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.384670    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:16.387052 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.387024    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d04e5de2-0045-40dc-b5cf-99533e3724c2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk8rn\" (UID: \"d04e5de2-0045-40dc-b5cf-99533e3724c2\") " pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:16.408032 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.408008    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk8rn"
Oct 31 14:46:16.600461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.600432    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"]
Oct 31 14:46:16.606146 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.606121    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:16.686470 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.686394    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/663b9426-03ca-4551-b31a-3a5039707fa9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9kpgn\" (UID: \"663b9426-03ca-4551-b31a-3a5039707fa9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:16.708438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.708400    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 14:46:16.708563 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.708471    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-glp2g\""
Oct 31 14:46:16.709165 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.709135    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"]
Oct 31 14:46:16.777207 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.777180    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk8rn"]
Oct 31 14:46:16.780829 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:16.780802    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04e5de2_0045_40dc_b5cf_99533e3724c2.slice/crio-67fb04c3045ebbe40fddc2c16f12279727c4a143a90be42f6cf7f588d06c64a4 WatchSource:0}: Error finding container 67fb04c3045ebbe40fddc2c16f12279727c4a143a90be42f6cf7f588d06c64a4: Status 404 returned error can't find the container with id 67fb04c3045ebbe40fddc2c16f12279727c4a143a90be42f6cf7f588d06c64a4
Oct 31 14:46:16.787751 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.787724    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/663b9426-03ca-4551-b31a-3a5039707fa9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9kpgn\" (UID: \"663b9426-03ca-4551-b31a-3a5039707fa9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:16.790298 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.790195    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/663b9426-03ca-4551-b31a-3a5039707fa9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-9kpgn\" (UID: \"663b9426-03ca-4551-b31a-3a5039707fa9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:16.915919 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:16.915887    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:17.278068 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.278044    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"]
Oct 31 14:46:17.281404 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:17.281377    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663b9426_03ca_4551_b31a_3a5039707fa9.slice/crio-17687d8e0f371b717f054eea541c05a53cf115047686ac369ad36434e0b08cf5 WatchSource:0}: Error finding container 17687d8e0f371b717f054eea541c05a53cf115047686ac369ad36434e0b08cf5: Status 404 returned error can't find the container with id 17687d8e0f371b717f054eea541c05a53cf115047686ac369ad36434e0b08cf5
Oct 31 14:46:17.506880 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.506852    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ls8dq"
Oct 31 14:46:17.533421 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.533350    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn" event={"ID":"663b9426-03ca-4551-b31a-3a5039707fa9","Type":"ContainerStarted","Data":"17687d8e0f371b717f054eea541c05a53cf115047686ac369ad36434e0b08cf5"}
Oct 31 14:46:17.534767 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.534742    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk8rn" event={"ID":"d04e5de2-0045-40dc-b5cf-99533e3724c2","Type":"ContainerStarted","Data":"ca4ff5839f142d07caadf13f03007bfd59bd1baebe446c23edf02b9ad930844c"}
Oct 31 14:46:17.534858 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.534786    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk8rn" event={"ID":"d04e5de2-0045-40dc-b5cf-99533e3724c2","Type":"ContainerStarted","Data":"67fb04c3045ebbe40fddc2c16f12279727c4a143a90be42f6cf7f588d06c64a4"}
Oct 31 14:46:17.985046 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:17.985022    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4sfm"
Oct 31 14:46:18.539860 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:18.539827    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk8rn" event={"ID":"d04e5de2-0045-40dc-b5cf-99533e3724c2","Type":"ContainerStarted","Data":"1919153050df15d8ba3fd242ea22f9aa241ad225c065aaf0c3e9ff8b013575bd"}
Oct 31 14:46:19.552127 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:19.552088    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn" event={"ID":"663b9426-03ca-4551-b31a-3a5039707fa9","Type":"ContainerStarted","Data":"0bf3ebff654c2b6facd402de046794169498b3769077a025ef4f1289a0df9f54"}
Oct 31 14:46:19.552618 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:19.552315    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:19.558809 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:19.558782    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn"
Oct 31 14:46:19.665061 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:19.665006    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-9kpgn" podStartSLOduration=2.308851713 podStartE2EDuration="3.664992831s" totalImagesPullingTime="1.356141118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:19.663911105 +0000 UTC m=+20.390192082" watchObservedRunningTime="2026-10-31 14:46:19.664992831 +0000 UTC m=+20.391273840"
Oct 31 14:46:20.557358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.557319    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk8rn" event={"ID":"d04e5de2-0045-40dc-b5cf-99533e3724c2","Type":"ContainerStarted","Data":"5691ffc04230b63dc7dfafcb09cb28d415bad7e622621249f19daf335b1042d3"}
Oct 31 14:46:20.659961 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.659928    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:46:20.661726 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.661711    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710287 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710250    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:46:20.710498 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710478    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710585 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710506    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710585 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710536    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710585 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710552    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710730 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710635    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9knm\" (UniqueName: \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710799 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710779    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.710837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.710821    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.722984 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722921    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 14:46:20.722984 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722946    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 14:46:20.722984 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722965    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 14:46:20.722984 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722947    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jngf7\""
Oct 31 14:46:20.723223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722973    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 14:46:20.723223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722997    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 14:46:20.723223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.722996    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 14:46:20.723223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.723053    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 14:46:20.812105 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812029    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812105 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812076    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812105 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812094    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812385 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812120    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812385 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812137    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812385 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812156    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9knm\" (UniqueName: \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812385 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812190    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812652 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812632    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.812938 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.812912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.813060 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.813041    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.813098 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.813046    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.814802 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.814776    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.814989 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.814974    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.869121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.869093    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9knm\" (UniqueName: \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\") pod \"console-6db59ccdd5-29tbc\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") " pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:20.971260 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:20.971228    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:21.211602 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.211499    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"]
Oct 31 14:46:21.213478 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.213462    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.265298 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.265272    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 14:46:21.265545 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.265533    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 14:46:21.265747 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.265731    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j72m7\""
Oct 31 14:46:21.265789 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.265767    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 14:46:21.265939 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.265926    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 14:46:21.266080 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.266067    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 14:46:21.315352 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.315323    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"]
Oct 31 14:46:21.315891 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.315871    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.315945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.315910    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.315977 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.315943    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba08ded1-719d-4f0c-826a-d5b47b06eded-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.316090 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.316072    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltv4\" (UniqueName: \"kubernetes.io/projected/ba08ded1-719d-4f0c-826a-d5b47b06eded-kube-api-access-9ltv4\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.386362 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.386323    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:46:21.389995 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:21.389968    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a91f700_90f1_4e51_94d9_d7d2320fc16e.slice/crio-ad0af696d578def7dfb33a8dc6acada1ebaa486844c88e3a6cf1b7f00f4cc68d WatchSource:0}: Error finding container ad0af696d578def7dfb33a8dc6acada1ebaa486844c88e3a6cf1b7f00f4cc68d: Status 404 returned error can't find the container with id ad0af696d578def7dfb33a8dc6acada1ebaa486844c88e3a6cf1b7f00f4cc68d
Oct 31 14:46:21.417125 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.417095    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltv4\" (UniqueName: \"kubernetes.io/projected/ba08ded1-719d-4f0c-826a-d5b47b06eded-kube-api-access-9ltv4\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.417232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.417133    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.417232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.417172    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.417232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.417202    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba08ded1-719d-4f0c-826a-d5b47b06eded-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.417857 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.417835    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba08ded1-719d-4f0c-826a-d5b47b06eded-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.419608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.419588    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.419696 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.419679    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba08ded1-719d-4f0c-826a-d5b47b06eded-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.473600 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.473519    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltv4\" (UniqueName: \"kubernetes.io/projected/ba08ded1-719d-4f0c-826a-d5b47b06eded-kube-api-access-9ltv4\") pod \"prometheus-operator-9cd89b74c-xncbp\" (UID: \"ba08ded1-719d-4f0c-826a-d5b47b06eded\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.524741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.524711    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"
Oct 31 14:46:21.562396 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.562358    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db59ccdd5-29tbc" event={"ID":"5a91f700-90f1-4e51-94d9-d7d2320fc16e","Type":"ContainerStarted","Data":"ad0af696d578def7dfb33a8dc6acada1ebaa486844c88e3a6cf1b7f00f4cc68d"}
Oct 31 14:46:21.889718 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:21.889671    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-xncbp"]
Oct 31 14:46:21.896816 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:21.896787    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba08ded1_719d_4f0c_826a_d5b47b06eded.slice/crio-98954798de733ee4b9d0ea6ad02890308dbe87e0122ec8b9ea7690726470627f WatchSource:0}: Error finding container 98954798de733ee4b9d0ea6ad02890308dbe87e0122ec8b9ea7690726470627f: Status 404 returned error can't find the container with id 98954798de733ee4b9d0ea6ad02890308dbe87e0122ec8b9ea7690726470627f
Oct 31 14:46:22.567259 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:22.567221    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp" event={"ID":"ba08ded1-719d-4f0c-826a-d5b47b06eded","Type":"ContainerStarted","Data":"98954798de733ee4b9d0ea6ad02890308dbe87e0122ec8b9ea7690726470627f"}
Oct 31 14:46:23.388688 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:23.388650    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktm59"
Oct 31 14:46:23.573421 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:23.573393    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp" event={"ID":"ba08ded1-719d-4f0c-826a-d5b47b06eded","Type":"ContainerStarted","Data":"2f000237c1dd5d8b73971be017df4ca2d8987b6f1cd0357cfce1f1a6394f736c"}
Oct 31 14:46:24.580087 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:24.580048    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp" event={"ID":"ba08ded1-719d-4f0c-826a-d5b47b06eded","Type":"ContainerStarted","Data":"81429dfe7faac46c1a1a64142da20f27df40fc897f7407d915d29e1ce4178704"}
Oct 31 14:46:24.696721 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:24.696664    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-xncbp" podStartSLOduration=2.098820351 podStartE2EDuration="3.696648877s" totalImagesPullingTime="1.597828526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:24.695515824 +0000 UTC m=+25.421796802" watchObservedRunningTime="2026-10-31 14:46:24.696648877 +0000 UTC m=+25.422929854"
Oct 31 14:46:25.585039 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:25.585003    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db59ccdd5-29tbc" event={"ID":"5a91f700-90f1-4e51-94d9-d7d2320fc16e","Type":"ContainerStarted","Data":"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"}
Oct 31 14:46:25.703295 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:25.703168    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6db59ccdd5-29tbc" podStartSLOduration=2.009230764 podStartE2EDuration="5.703150151s" totalImagesPullingTime="3.693919387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:25.702343332 +0000 UTC m=+26.428624311" watchObservedRunningTime="2026-10-31 14:46:25.703150151 +0000 UTC m=+26.429431126"
Oct 31 14:46:27.812924 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.812895    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"]
Oct 31 14:46:27.849202 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.849171    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.862658 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.862633    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xmtt5"]
Oct 31 14:46:27.877031 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.877005    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.919431 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.919394    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"]
Oct 31 14:46:27.953740 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.953711    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 14:46:27.953740 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.953720    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 14:46:27.960959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.960941    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8c6841a-9e72-4b90-a387-012c684b5e2f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.960968    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqht\" (UniqueName: \"kubernetes.io/projected/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-api-access-jxqht\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.960983    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961005    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-textfile\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961021    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961105    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961158    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-sys\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961256 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961203    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961256 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961237    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-wtmp\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961360 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961262    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961360 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961330    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-root\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961368    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tw8b\" (UniqueName: \"kubernetes.io/projected/202b35c4-ef2a-42e3-b975-233bdcb69bdb-kube-api-access-4tw8b\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961404    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:27.961463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961453    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-metrics-client-ca\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:27.961554 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:27.961495    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-tls\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.001457 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001425    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 14:46:28.001668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001512    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 14:46:28.001668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001520    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 14:46:28.001668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001530    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 14:46:28.001668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001606    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l5cfh\""
Oct 31 14:46:28.001668 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.001665    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\""
Oct 31 14:46:28.062244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062213    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-sys\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062247    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062274    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-wtmp\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062306    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062333    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-sys\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062339    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-root\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062376    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-root\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062390    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tw8b\" (UniqueName: \"kubernetes.io/projected/202b35c4-ef2a-42e3-b975-233bdcb69bdb-kube-api-access-4tw8b\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062419    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062433    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-wtmp\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062440    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-metrics-client-ca\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062493    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-tls\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062535    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8c6841a-9e72-4b90-a387-012c684b5e2f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062595    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqht\" (UniqueName: \"kubernetes.io/projected/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-api-access-jxqht\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062622    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062658    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-textfile\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062682    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.062900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062716    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.063271 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.062981    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d8c6841a-9e72-4b90-a387-012c684b5e2f-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.063271 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.063259    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.063381 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.063271    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202b35c4-ef2a-42e3-b975-233bdcb69bdb-metrics-client-ca\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.063381 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.063259    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.063669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.063432    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6841a-9e72-4b90-a387-012c684b5e2f-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.063669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.063467    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-textfile\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.065418 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.065392    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-tls\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.065564 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.065403    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.065863 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.065679    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202b35c4-ef2a-42e3-b975-233bdcb69bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.065863 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.065683    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.119888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.119860    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tw8b\" (UniqueName: \"kubernetes.io/projected/202b35c4-ef2a-42e3-b975-233bdcb69bdb-kube-api-access-4tw8b\") pod \"node-exporter-xmtt5\" (UID: \"202b35c4-ef2a-42e3-b975-233bdcb69bdb\") " pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.120245 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.120223    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqht\" (UniqueName: \"kubernetes.io/projected/d8c6841a-9e72-4b90-a387-012c684b5e2f-kube-api-access-jxqht\") pod \"kube-state-metrics-5dc8d7d89b-jqxph\" (UID: \"d8c6841a-9e72-4b90-a387-012c684b5e2f\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.197760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.197730    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"
Oct 31 14:46:28.239440 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.239413    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xmtt5"
Oct 31 14:46:28.262548 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:28.262516    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b35c4_ef2a_42e3_b975_233bdcb69bdb.slice/crio-780c2cfb00c17822fdf403b74110b427763630bdb4b0595ca1785ecbcae1fd3e WatchSource:0}: Error finding container 780c2cfb00c17822fdf403b74110b427763630bdb4b0595ca1785ecbcae1fd3e: Status 404 returned error can't find the container with id 780c2cfb00c17822fdf403b74110b427763630bdb4b0595ca1785ecbcae1fd3e
Oct 31 14:46:28.595490 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.595452    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmtt5" event={"ID":"202b35c4-ef2a-42e3-b975-233bdcb69bdb","Type":"ContainerStarted","Data":"780c2cfb00c17822fdf403b74110b427763630bdb4b0595ca1785ecbcae1fd3e"}
Oct 31 14:46:28.649749 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:28.649722    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph"]
Oct 31 14:46:28.652645 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:28.652621    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c6841a_9e72_4b90_a387_012c684b5e2f.slice/crio-b47254231f7d81068dcf49dc67db6381c17e5ebcd283587b5f701e8e1eef015a WatchSource:0}: Error finding container b47254231f7d81068dcf49dc67db6381c17e5ebcd283587b5f701e8e1eef015a: Status 404 returned error can't find the container with id b47254231f7d81068dcf49dc67db6381c17e5ebcd283587b5f701e8e1eef015a
Oct 31 14:46:29.600655 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:29.600613    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph" event={"ID":"d8c6841a-9e72-4b90-a387-012c684b5e2f","Type":"ContainerStarted","Data":"b47254231f7d81068dcf49dc67db6381c17e5ebcd283587b5f701e8e1eef015a"}
Oct 31 14:46:30.077803 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.077775    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:46:30.097624 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.097602    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.134203 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.134164    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:46:30.153122 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.153090    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 14:46:30.158244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.158221    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 14:46:30.180643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.180607    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.180643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.180646    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.180832 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.180666    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.180875 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.180850    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.180914 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.180895    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181214 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181181    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181230    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181282    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181313    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181381    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181461    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181482    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.181517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.181498    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dss7\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.182911 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.182892    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 14:46:30.282356 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282312    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282356 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282364    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282396    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282426    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282434    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282456    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282499    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282544    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dss7\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282601    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282623    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gfp4g\""
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282626    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282640    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282651    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282701    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282712    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 14:46:30.282795 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.282727    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.283756 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.283000    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 14:46:30.283756 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.283058    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 14:46:30.283756 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.283523    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.284514 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.284485    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 14:46:30.285333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.284799    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.285333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.284817    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.287422 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.287398    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.287656 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.287634    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.287912 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.287880    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.288012 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.287912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.288320 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.288278    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.288414 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.288393    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.288780 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.288756    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.289168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.289144    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.289301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.289286    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.342869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.342798    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dss7\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\") pod \"alertmanager-main-0\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.508120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.508086    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:46:30.605382 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.605309    4549 generic.go:356] "Generic (PLEG): container finished" podID="202b35c4-ef2a-42e3-b975-233bdcb69bdb" containerID="fa3fe7afa12210f79dfde4a3959cb712804957992ba5191bd18a6da765945005" exitCode=0
Oct 31 14:46:30.605794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.605398    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmtt5" event={"ID":"202b35c4-ef2a-42e3-b975-233bdcb69bdb","Type":"ContainerDied","Data":"fa3fe7afa12210f79dfde4a3959cb712804957992ba5191bd18a6da765945005"}
Oct 31 14:46:30.956267 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.956235    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:46:30.970949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.970925    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:30.971302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.971276    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:30.971355 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.971330    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:30.977224 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:30.977193    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:31.020501 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.020462    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:46:31.024111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.024077    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 14:46:31.090793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.090765    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.090950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.090805    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.090950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.090937    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdtk\" (UniqueName: \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.091071 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.090966    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.091071 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.090992    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.091176 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.091119    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.091230 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.091174    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.091230 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.091211    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.124973 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.124614    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:46:31.127272 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:31.127242    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c951f1d_c2d6_49f3_80e1_d9b1be5e8095.slice/crio-77c1156d518f8225fce00323e891877f070b359d5c903438f7e0ee4db533f493 WatchSource:0}: Error finding container 77c1156d518f8225fce00323e891877f070b359d5c903438f7e0ee4db533f493: Status 404 returned error can't find the container with id 77c1156d518f8225fce00323e891877f070b359d5c903438f7e0ee4db533f493
Oct 31 14:46:31.192378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192340    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192374    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192398    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdtk\" (UniqueName: \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192421    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192443    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192485    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192513    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.192660 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.192542    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.193122 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.193088    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.193330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.193245    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.193330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.193282    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.193330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.193314    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.193718 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.193683    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.195180 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.195162    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.195254 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.195205    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.249794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.249762    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdtk\" (UniqueName: \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\") pod \"console-554bb6f69d-fdxpt\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") " pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.297564 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.297525    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:31.609895 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.609863    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"77c1156d518f8225fce00323e891877f070b359d5c903438f7e0ee4db533f493"}
Oct 31 14:46:31.611973 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.611953    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph" event={"ID":"d8c6841a-9e72-4b90-a387-012c684b5e2f","Type":"ContainerStarted","Data":"2cb3fd1a13fa699b43839ac79ff8f2f5be07e89910cd27dd17e1709b3dbddfda"}
Oct 31 14:46:31.612094 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.612001    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph" event={"ID":"d8c6841a-9e72-4b90-a387-012c684b5e2f","Type":"ContainerStarted","Data":"1a41f11c5f8be872ef463d1cdfbebc375ddc9740d2a0772aac6341edd29418ab"}
Oct 31 14:46:31.612094 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.612026    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph" event={"ID":"d8c6841a-9e72-4b90-a387-012c684b5e2f","Type":"ContainerStarted","Data":"a5aae744abae577820541b28d79d6b64b6ec3d322e22b66cb237df909c7a3239"}
Oct 31 14:46:31.614142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.614116    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmtt5" event={"ID":"202b35c4-ef2a-42e3-b975-233bdcb69bdb","Type":"ContainerStarted","Data":"f7a96b36c26816a063e1f0832821b358aca401e39da19ef06467987ca4979ba3"}
Oct 31 14:46:31.614260 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.614158    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xmtt5" event={"ID":"202b35c4-ef2a-42e3-b975-233bdcb69bdb","Type":"ContainerStarted","Data":"8203d85d38e2f699a09e1258ca5d5ad5bf4a7f88ba0c6c8307c75088412d04c5"}
Oct 31 14:46:31.617988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.617965    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:46:31.712390 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.712215    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:46:31.714843 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:31.714818    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79522e1_951b_4d01_a731_efec04bbb818.slice/crio-82d3c9f994e6b05ad92074be71a51a0758eb786f3f0ba8a9ce7a51a51eaebf9a WatchSource:0}: Error finding container 82d3c9f994e6b05ad92074be71a51a0758eb786f3f0ba8a9ce7a51a51eaebf9a: Status 404 returned error can't find the container with id 82d3c9f994e6b05ad92074be71a51a0758eb786f3f0ba8a9ce7a51a51eaebf9a
Oct 31 14:46:31.729015 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.728960    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jqxph" podStartSLOduration=2.716890784 podStartE2EDuration="4.728941901s" totalImagesPullingTime="2.012051117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:31.728157309 +0000 UTC m=+32.454438286" watchObservedRunningTime="2026-10-31 14:46:31.728941901 +0000 UTC m=+32.455222881"
Oct 31 14:46:31.963747 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.963637    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xmtt5" podStartSLOduration=3.079822921 podStartE2EDuration="4.963611894s" totalImagesPullingTime="1.789449017s" totalInitContainerRuntime="94.339956ms" isStatefulPod=true podCreationTimestamp="2026-10-31 14:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:31.963261358 +0000 UTC m=+32.689542337" watchObservedRunningTime="2026-10-31 14:46:31.963611894 +0000 UTC m=+32.689892876"
Oct 31 14:46:31.995425 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:31.995398    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ljbk9"
Oct 31 14:46:32.619146 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.619114    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554bb6f69d-fdxpt" event={"ID":"c79522e1-951b-4d01-a731-efec04bbb818","Type":"ContainerStarted","Data":"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"}
Oct 31 14:46:32.619589 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.619157    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554bb6f69d-fdxpt" event={"ID":"c79522e1-951b-4d01-a731-efec04bbb818","Type":"ContainerStarted","Data":"82d3c9f994e6b05ad92074be71a51a0758eb786f3f0ba8a9ce7a51a51eaebf9a"}
Oct 31 14:46:32.692257 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.692216    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5df946bc5-wfb69"]
Oct 31 14:46:32.703794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.703777    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.738074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.738028    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-554bb6f69d-fdxpt" podStartSLOduration=2.7380148010000003 podStartE2EDuration="2.738014801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:32.737331324 +0000 UTC m=+33.463612302" watchObservedRunningTime="2026-10-31 14:46:32.738014801 +0000 UTC m=+33.464295778"
Oct 31 14:46:32.752796 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.752773    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5df946bc5-wfb69"]
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755860    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755879    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-72qw8\""
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755887    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755902    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755938    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 14:46:32.756017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.755980    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-arg423tua776m\""
Oct 31 14:46:32.810982 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.810953    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/289504d6-1764-433c-b803-3ce22a49f070-audit-log\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811098 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.810989    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-client-certs\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811098 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.811006    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-metrics-server-audit-profiles\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.811092    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-client-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.811146    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-tls\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811231 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.811180    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.811231 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.811197    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgldt\" (UniqueName: \"kubernetes.io/projected/289504d6-1764-433c-b803-3ce22a49f070-kube-api-access-jgldt\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.911738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.911666    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-client-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.911738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.911711    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-tls\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.911905 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.911882    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.911945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.911917    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgldt\" (UniqueName: \"kubernetes.io/projected/289504d6-1764-433c-b803-3ce22a49f070-kube-api-access-jgldt\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.911989 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.911977    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/289504d6-1764-433c-b803-3ce22a49f070-audit-log\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.912038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.912009    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-client-certs\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.912087 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.912035    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-metrics-server-audit-profiles\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.912438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.912413    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/289504d6-1764-433c-b803-3ce22a49f070-audit-log\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.912817 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.912787    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.912953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.912870    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/289504d6-1764-433c-b803-3ce22a49f070-metrics-server-audit-profiles\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.914266 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.914248    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-tls\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.914373 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.914356    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-client-ca-bundle\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.914635 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.914617    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/289504d6-1764-433c-b803-3ce22a49f070-secret-metrics-server-client-certs\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:32.969232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:32.969208    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgldt\" (UniqueName: \"kubernetes.io/projected/289504d6-1764-433c-b803-3ce22a49f070-kube-api-access-jgldt\") pod \"metrics-server-5df946bc5-wfb69\" (UID: \"289504d6-1764-433c-b803-3ce22a49f070\") " pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:33.036213 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:33.036191    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:33.469330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:33.469260    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5df946bc5-wfb69"]
Oct 31 14:46:33.472087 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:33.472066    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289504d6_1764_433c_b803_3ce22a49f070.slice/crio-b936a44a36f7110d1a3de91650050c45938246b262aa22bf611108531ad7f353 WatchSource:0}: Error finding container b936a44a36f7110d1a3de91650050c45938246b262aa22bf611108531ad7f353: Status 404 returned error can't find the container with id b936a44a36f7110d1a3de91650050c45938246b262aa22bf611108531ad7f353
Oct 31 14:46:33.624188 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:33.624153    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709" exitCode=0
Oct 31 14:46:33.624614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:33.624239    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709"}
Oct 31 14:46:33.625580 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:33.625541    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69" event={"ID":"289504d6-1764-433c-b803-3ce22a49f070","Type":"ContainerStarted","Data":"b936a44a36f7110d1a3de91650050c45938246b262aa22bf611108531ad7f353"}
Oct 31 14:46:35.640366 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:35.640333    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"}
Oct 31 14:46:35.640716 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:35.640451    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"}
Oct 31 14:46:35.640716 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:35.640494    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"}
Oct 31 14:46:35.642741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:35.642715    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69" event={"ID":"289504d6-1764-433c-b803-3ce22a49f070","Type":"ContainerStarted","Data":"2cfdd4f2c4752cda5e768823e7eed6b384f9b313dc79a3bdcee18ed04a8ebc45"}
Oct 31 14:46:35.785062 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:35.785007    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69" podStartSLOduration=1.913083844 podStartE2EDuration="3.784994074s" totalImagesPullingTime="1.87191023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:35.784007753 +0000 UTC m=+36.510288733" watchObservedRunningTime="2026-10-31 14:46:35.784994074 +0000 UTC m=+36.511275052"
Oct 31 14:46:36.650537 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:36.650498    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"}
Oct 31 14:46:36.651154 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:36.650595    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"}
Oct 31 14:46:37.012617 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:46:37.012558    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:46:37.658049 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:37.658020    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerStarted","Data":"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"}
Oct 31 14:46:37.879681 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:37.878927    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.019329219 podStartE2EDuration="8.878907334s" totalImagesPullingTime="4.833909201s" totalInitContainerRuntime="25.668914ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:37.829024138 +0000 UTC m=+38.555305127" watchObservedRunningTime="2026-10-31 14:46:37.878907334 +0000 UTC m=+38.605188312"
Oct 31 14:46:38.645725 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.645694    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:46:38.677932 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.677900    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:46:38.684595 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.684548    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.757641 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.757603    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.763203    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88hb\" (UniqueName: \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.763291    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.763372    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.763738    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.764044    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.764069    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.764097    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.764461 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.764193    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865169    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865200    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865434 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865220    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865434 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865251    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865434 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865292    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88hb\" (UniqueName: \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865434 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865330    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865522    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865593    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.865674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.865655    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.866238 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.866104    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.866238 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.866110    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.866238 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.866110    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.866505 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.866336    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.868100 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.868081    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.868189 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.868082    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.924103 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.924036    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88hb\" (UniqueName: \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\") pod \"console-5d8b8b96d5-26g7h\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") " pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:38.999049 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:38.999019    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:39.374001 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:39.373973    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:46:39.377145 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:46:39.377117    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941f0c62_1072_4e25_ad99_3c725fa54a31.slice/crio-0c026fdbe5414fc8e656497efda896498d79b5633b7b710852643cbea74ed6dc WatchSource:0}: Error finding container 0c026fdbe5414fc8e656497efda896498d79b5633b7b710852643cbea74ed6dc: Status 404 returned error can't find the container with id 0c026fdbe5414fc8e656497efda896498d79b5633b7b710852643cbea74ed6dc
Oct 31 14:46:39.667332 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:39.667233    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b8b96d5-26g7h" event={"ID":"941f0c62-1072-4e25-ad99-3c725fa54a31","Type":"ContainerStarted","Data":"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"}
Oct 31 14:46:39.667332 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:39.667305    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b8b96d5-26g7h" event={"ID":"941f0c62-1072-4e25-ad99-3c725fa54a31","Type":"ContainerStarted","Data":"0c026fdbe5414fc8e656497efda896498d79b5633b7b710852643cbea74ed6dc"}
Oct 31 14:46:39.788031 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:39.786765    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d8b8b96d5-26g7h" podStartSLOduration=1.786748905 podStartE2EDuration="1.786748905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:39.784706645 +0000 UTC m=+40.510987624" watchObservedRunningTime="2026-10-31 14:46:39.786748905 +0000 UTC m=+40.513029884"
Oct 31 14:46:41.297723 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:41.297646    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:46:42.931923 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:42.931894    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:46:42.981866 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:42.981840    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:46:42.985462 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:42.985446    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.042950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.042922    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:46:43.103741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103709    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.103741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103743    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.103964 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103762    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.103964 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103783    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.103964 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103851    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.103964 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103933    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jd6d\" (UniqueName: \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.104131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.103968    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.104131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.104049    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205055    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205092    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205109    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205219    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205254    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205282    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jd6d\" (UniqueName: \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205316    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205587 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205386    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.205587 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.205516    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.206103 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.206058    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.206103 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.206095    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.206282 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.206262    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.206342 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.206325    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.207727 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.207708    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.207812 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.207795    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.268810 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.268775    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jd6d\" (UniqueName: \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\") pod \"console-666fbc7699-chgcd\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") " pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.299646 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.299613    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:43.670289 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.670260    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:46:43.685679 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:43.685611    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fbc7699-chgcd" event={"ID":"73e64d5d-20c3-498c-9de4-227282d4df67","Type":"ContainerStarted","Data":"82da1b0d0c0b419459cd4e1a28ab66a04c6ef68fdb5f2556afaf2220880a8668"}
Oct 31 14:46:44.690497 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:44.690453    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fbc7699-chgcd" event={"ID":"73e64d5d-20c3-498c-9de4-227282d4df67","Type":"ContainerStarted","Data":"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"}
Oct 31 14:46:44.857197 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:44.857135    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-666fbc7699-chgcd" podStartSLOduration=2.85711996 podStartE2EDuration="2.85711996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:44.806907879 +0000 UTC m=+45.533188870" watchObservedRunningTime="2026-10-31 14:46:44.85711996 +0000 UTC m=+45.583400938"
Oct 31 14:46:48.999444 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:48.999401    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:46:53.036986 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.036949    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:53.037441 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.037011    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:46:53.299903 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.299814    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:53.299903 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.299880    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:53.304385 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.304365    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:53.725967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.725891    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:46:53.891356 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:46:53.891326    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:47:03.667012 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.666967    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-554bb6f69d-fdxpt" podUID="c79522e1-951b-4d01-a731-efec04bbb818" containerName="console" containerID="cri-o://af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad" gracePeriod=15
Oct 31 14:47:03.925564 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.925511    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:47:03.966111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966083    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\" (UniqueName: \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966115    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966307 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966160    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966307 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966204    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966307 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966252    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966307 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966285    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966319    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966359    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\") pod \"c79522e1-951b-4d01-a731-efec04bbb818\" (UID: \"c79522e1-951b-4d01-a731-efec04bbb818\") "
Oct 31 14:47:03.966482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966449    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:03.966870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966666    4549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c79522e1-951b-4d01-a731-efec04bbb818-tmp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:03.966870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966631    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:03.966870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966744    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:03.966870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966835    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:03.967043 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.966911    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:03.968725 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.968702    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "kube-api-access-drdtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:03.968810 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.968770    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:03.968887 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:03.968869    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config" pod "c79522e1-951b-4d01-a731-efec04bbb818" (UID: "c79522e1-951b-4d01-a731-efec04bbb818"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:04.067855 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067824    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdtk\" (UniqueName: \"kubernetes.io/projected/c79522e1-951b-4d01-a731-efec04bbb818-kube-api-access-drdtk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.067855 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067854    4549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-oauth-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.068058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067867    4549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-trusted-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.068058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067880    4549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-oauth-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.068058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067891    4549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-console-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.068058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067902    4549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c79522e1-951b-4d01-a731-efec04bbb818-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.068058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.067914    4549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c79522e1-951b-4d01-a731-efec04bbb818-console-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:04.763824 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.763788    4549 generic.go:356] "Generic (PLEG): container finished" podID="c79522e1-951b-4d01-a731-efec04bbb818" containerID="af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad" exitCode=0
Oct 31 14:47:04.764299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.763839    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554bb6f69d-fdxpt"
Oct 31 14:47:04.764299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.763880    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554bb6f69d-fdxpt" event={"ID":"c79522e1-951b-4d01-a731-efec04bbb818","Type":"ContainerDied","Data":"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"}
Oct 31 14:47:04.764299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.763939    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554bb6f69d-fdxpt" event={"ID":"c79522e1-951b-4d01-a731-efec04bbb818","Type":"ContainerDied","Data":"82d3c9f994e6b05ad92074be71a51a0758eb786f3f0ba8a9ce7a51a51eaebf9a"}
Oct 31 14:47:04.764299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.763967    4549 scope.go:122] "RemoveContainer" containerID="af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"
Oct 31 14:47:04.773149 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.773131    4549 scope.go:122] "RemoveContainer" containerID="af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"
Oct 31 14:47:04.773433 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:04.773416    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad\": container with ID starting with af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad not found: ID does not exist" containerID="af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"
Oct 31 14:47:04.773498 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.773444    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad"} err="failed to get container status \"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad\": rpc error: code = NotFound desc = could not find container \"af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad\": container with ID starting with af54907fa89c2e6bb860eda0ce2d9dec69728c7408d79178188aca73f1eb35ad not found: ID does not exist"
Oct 31 14:47:04.926304 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.926269    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:47:04.976072 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:04.976042    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-554bb6f69d-fdxpt"]
Oct 31 14:47:05.419200 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:05.419166    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79522e1-951b-4d01-a731-efec04bbb818" path="/var/lib/kubelet/pods/c79522e1-951b-4d01-a731-efec04bbb818/volumes"
Oct 31 14:47:07.010151 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:07.010110    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:47:07.951944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:07.951904    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d8b8b96d5-26g7h" podUID="941f0c62-1072-4e25-ad99-3c725fa54a31" containerName="console" containerID="cri-o://7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679" gracePeriod=15
Oct 31 14:47:08.233534 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.233465    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:47:08.300720 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300689    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.300869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300727    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.300869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300754    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\" (UniqueName: \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.300869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300780    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.300869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300817    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.300869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300863    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.301117 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.300947    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\" (UniqueName: \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.301117 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301019    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\") pod \"941f0c62-1072-4e25-ad99-3c725fa54a31\" (UID: \"941f0c62-1072-4e25-ad99-3c725fa54a31\") "
Oct 31 14:47:08.301219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301194    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:08.301219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301201    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:08.301219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301206    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:08.301378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301314    4549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/941f0c62-1072-4e25-ad99-3c725fa54a31-tmp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.301378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301330    4549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-oauth-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.301378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301340    4549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-trusted-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.301536 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301417    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:08.301621 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.301484    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:08.303127 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.303102    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:08.303254 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.303157    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:08.303254 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.303243    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb" pod "941f0c62-1072-4e25-ad99-3c725fa54a31" (UID: "941f0c62-1072-4e25-ad99-3c725fa54a31"). InnerVolumeSpecName "kube-api-access-z88hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:08.401887 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.401842    4549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.401887 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.401882    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88hb\" (UniqueName: \"kubernetes.io/projected/941f0c62-1072-4e25-ad99-3c725fa54a31-kube-api-access-z88hb\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.401887 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.401895    4549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-oauth-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.402131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.401908    4549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/941f0c62-1072-4e25-ad99-3c725fa54a31-console-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.402131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.401921    4549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/941f0c62-1072-4e25-ad99-3c725fa54a31-console-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:08.782221 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.782186    4549 generic.go:356] "Generic (PLEG): container finished" podID="941f0c62-1072-4e25-ad99-3c725fa54a31" containerID="7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679" exitCode=0
Oct 31 14:47:08.782424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.782237    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d8b8b96d5-26g7h"
Oct 31 14:47:08.782424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.782249    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b8b96d5-26g7h" event={"ID":"941f0c62-1072-4e25-ad99-3c725fa54a31","Type":"ContainerDied","Data":"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"}
Oct 31 14:47:08.782424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.782287    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d8b8b96d5-26g7h" event={"ID":"941f0c62-1072-4e25-ad99-3c725fa54a31","Type":"ContainerDied","Data":"0c026fdbe5414fc8e656497efda896498d79b5633b7b710852643cbea74ed6dc"}
Oct 31 14:47:08.782424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.782306    4549 scope.go:122] "RemoveContainer" containerID="7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"
Oct 31 14:47:08.790712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.790691    4549 scope.go:122] "RemoveContainer" containerID="7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"
Oct 31 14:47:08.790970 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:08.790950    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679\": container with ID starting with 7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679 not found: ID does not exist" containerID="7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"
Oct 31 14:47:08.791016 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.790981    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679"} err="failed to get container status \"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679\": rpc error: code = NotFound desc = could not find container \"7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679\": container with ID starting with 7048ef0945902ae3626057c26acacc4e44894d1d933362d5ae53e5e403ec8679 not found: ID does not exist"
Oct 31 14:47:08.944871 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.944840    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:47:08.994410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:08.994366    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d8b8b96d5-26g7h"]
Oct 31 14:47:09.416113 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:09.416081    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="941f0c62-1072-4e25-ad99-3c725fa54a31" path="/var/lib/kubelet/pods/941f0c62-1072-4e25-ad99-3c725fa54a31/volumes"
Oct 31 14:47:10.844848 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:10.844817    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:47:10.845321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:10.845008    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" podUID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" containerName="insights-gathering" containerID="cri-o://e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe" gracePeriod=30
Oct 31 14:47:11.084227 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.084202    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:47:11.122433 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122360    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") pod \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") "
Oct 31 14:47:11.122433 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122418    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") pod \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") "
Oct 31 14:47:11.122648 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122458    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") pod \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") "
Oct 31 14:47:11.122648 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122478    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") pod \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\" (UID: \"c1bd6a76-b8c7-45e9-8d16-678fd598eecb\") "
Oct 31 14:47:11.122880 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122821    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path" pod "c1bd6a76-b8c7-45e9-8d16-678fd598eecb" (UID: "c1bd6a76-b8c7-45e9-8d16-678fd598eecb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:11.122880 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122863    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle" pod "c1bd6a76-b8c7-45e9-8d16-678fd598eecb" (UID: "c1bd6a76-b8c7-45e9-8d16-678fd598eecb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:11.123007 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.122900    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle" pod "c1bd6a76-b8c7-45e9-8d16-678fd598eecb" (UID: "c1bd6a76-b8c7-45e9-8d16-678fd598eecb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:11.125209 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.125184    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd" pod "c1bd6a76-b8c7-45e9-8d16-678fd598eecb" (UID: "c1bd6a76-b8c7-45e9-8d16-678fd598eecb"). InnerVolumeSpecName "kube-api-access-wjfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:11.223326 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.223295    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjfhd\" (UniqueName: \"kubernetes.io/projected/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-kube-api-access-wjfhd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:11.223326 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.223325    4549 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-archives-path\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:11.223326 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.223335    4549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-trusted-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:11.223553 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.223344    4549 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1bd6a76-b8c7-45e9-8d16-678fd598eecb-service-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:11.794613 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.794561    4549 generic.go:356] "Generic (PLEG): container finished" podID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" containerID="e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe" exitCode=2
Oct 31 14:47:11.794613 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.794617    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-k4hdw-tw97l"
Oct 31 14:47:11.794877 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.794636    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" event={"ID":"c1bd6a76-b8c7-45e9-8d16-678fd598eecb","Type":"ContainerDied","Data":"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"}
Oct 31 14:47:11.794877 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.794685    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-k4hdw-tw97l" event={"ID":"c1bd6a76-b8c7-45e9-8d16-678fd598eecb","Type":"ContainerDied","Data":"c781b4039b9c4a22ee7809235b419a1c4ff417cb628070d959ee4495936b9d47"}
Oct 31 14:47:11.794877 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.794709    4549 scope.go:122] "RemoveContainer" containerID="e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"
Oct 31 14:47:11.802813 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.802795    4549 scope.go:122] "RemoveContainer" containerID="e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"
Oct 31 14:47:11.803094 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:11.803072    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe\": container with ID starting with e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe not found: ID does not exist" containerID="e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"
Oct 31 14:47:11.803161 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.803105    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe"} err="failed to get container status \"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe\": rpc error: code = NotFound desc = could not find container \"e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe\": container with ID starting with e9c225d6463aae908c48530498a68e7f1a230f89dd777eeff237d19f5e7ad5fe not found: ID does not exist"
Oct 31 14:47:11.955907 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:11.955872    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:47:12.005699 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:12.005660    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-k4hdw-tw97l"]
Oct 31 14:47:13.041967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:13.041940    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:47:13.045638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:13.045617    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5df946bc5-wfb69"
Oct 31 14:47:13.420367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:13.420282    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" path="/var/lib/kubelet/pods/c1bd6a76-b8c7-45e9-8d16-678fd598eecb/volumes"
Oct 31 14:47:17.927043 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927004    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75b695586c-5g7g5"]
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927455    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c79522e1-951b-4d01-a731-efec04bbb818" containerName="console"
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927474    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79522e1-951b-4d01-a731-efec04bbb818" containerName="console"
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927495    4549 cpu_manager.go:414] "containerMap: removing container" podUID="941f0c62-1072-4e25-ad99-3c725fa54a31" containerName="console"
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927502    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="941f0c62-1072-4e25-ad99-3c725fa54a31" containerName="console"
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927515    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" containerName="insights-gathering"
Oct 31 14:47:17.927546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927522    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" containerName="insights-gathering"
Oct 31 14:47:17.927878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927617    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79522e1-951b-4d01-a731-efec04bbb818" containerName="console"
Oct 31 14:47:17.927878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927631    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="941f0c62-1072-4e25-ad99-3c725fa54a31" containerName="console"
Oct 31 14:47:17.927878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.927642    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bd6a76-b8c7-45e9-8d16-678fd598eecb" containerName="insights-gathering"
Oct 31 14:47:17.931818 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.931795    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976164 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976136    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-oauth-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976172    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-trusted-ca-bundle\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976262    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976285    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/595398ef-1551-41c2-8b39-6b52a4dd4dba-tmp\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976301    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-service-ca\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976318    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqsx\" (UniqueName: \"kubernetes.io/projected/595398ef-1551-41c2-8b39-6b52a4dd4dba-kube-api-access-4hqsx\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976481 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976344    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:17.976514 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:17.976488    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-oauth-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.028299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.028264    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b695586c-5g7g5"]
Oct 31 14:47:18.076837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076803    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-oauth-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076844    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-oauth-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076879    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-trusted-ca-bundle\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076935    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076959    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/595398ef-1551-41c2-8b39-6b52a4dd4dba-tmp\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.076981    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-service-ca\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.077005    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqsx\" (UniqueName: \"kubernetes.io/projected/595398ef-1551-41c2-8b39-6b52a4dd4dba-kube-api-access-4hqsx\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077339 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.077043    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077398 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.077333    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/595398ef-1551-41c2-8b39-6b52a4dd4dba-tmp\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077843 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.077675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-oauth-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.077843 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.077749    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.078080 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.078060    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-trusted-ca-bundle\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.078170 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.078147    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/595398ef-1551-41c2-8b39-6b52a4dd4dba-service-ca\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.079494 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.079470    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-oauth-config\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.079678 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.079659    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/595398ef-1551-41c2-8b39-6b52a4dd4dba-console-serving-cert\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.133926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.133898    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqsx\" (UniqueName: \"kubernetes.io/projected/595398ef-1551-41c2-8b39-6b52a4dd4dba-kube-api-access-4hqsx\") pod \"console-75b695586c-5g7g5\" (UID: \"595398ef-1551-41c2-8b39-6b52a4dd4dba\") " pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.242347 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.242266    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:18.612864 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.612833    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b695586c-5g7g5"]
Oct 31 14:47:18.616160 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:47:18.616138    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595398ef_1551_41c2_8b39_6b52a4dd4dba.slice/crio-c8755c5199e3698fc680a3d4282479999003970e23b37bee7fdf1598a2e9bc45 WatchSource:0}: Error finding container c8755c5199e3698fc680a3d4282479999003970e23b37bee7fdf1598a2e9bc45: Status 404 returned error can't find the container with id c8755c5199e3698fc680a3d4282479999003970e23b37bee7fdf1598a2e9bc45
Oct 31 14:47:18.821766 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.821737    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b695586c-5g7g5" event={"ID":"595398ef-1551-41c2-8b39-6b52a4dd4dba","Type":"ContainerStarted","Data":"c0b1669975ed455688530735e89cafd826a7240212f564016184f21e512f70e5"}
Oct 31 14:47:18.821938 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.821785    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b695586c-5g7g5" event={"ID":"595398ef-1551-41c2-8b39-6b52a4dd4dba","Type":"ContainerStarted","Data":"c8755c5199e3698fc680a3d4282479999003970e23b37bee7fdf1598a2e9bc45"}
Oct 31 14:47:18.911565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.911484    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6db59ccdd5-29tbc" podUID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" containerName="console" containerID="cri-o://8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae" gracePeriod=15
Oct 31 14:47:18.936192 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:18.936134    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75b695586c-5g7g5" podStartSLOduration=1.936116272 podStartE2EDuration="1.936116272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:47:18.934974331 +0000 UTC m=+79.661255312" watchObservedRunningTime="2026-10-31 14:47:18.936116272 +0000 UTC m=+79.662397251"
Oct 31 14:47:19.158823 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.158802    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:47:19.185388 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185329    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185388 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185383    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185409    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185442    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\" (UniqueName: \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185468    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185498    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185521    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\") pod \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\" (UID: \"5a91f700-90f1-4e51-94d9-d7d2320fc16e\") "
Oct 31 14:47:19.185835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185812    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:19.185968 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185944    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:19.185968 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185956    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:19.186156 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.185944    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:19.187784 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.187761    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:19.187870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.187780    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "kube-api-access-r9knm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:19.187870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.187779    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config" pod "5a91f700-90f1-4e51-94d9-d7d2320fc16e" (UID: "5a91f700-90f1-4e51-94d9-d7d2320fc16e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:19.286743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286708    4549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.286743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286734    4549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-oauth-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.286743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286745    4549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.286743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286754    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9knm\" (UniqueName: \"kubernetes.io/projected/5a91f700-90f1-4e51-94d9-d7d2320fc16e-kube-api-access-r9knm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.287083 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286764    4549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a91f700-90f1-4e51-94d9-d7d2320fc16e-tmp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.287083 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286773    4549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a91f700-90f1-4e51-94d9-d7d2320fc16e-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.287083 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.286781    4549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a91f700-90f1-4e51-94d9-d7d2320fc16e-console-oauth-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:19.830382 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.830346    4549 generic.go:356] "Generic (PLEG): container finished" podID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" containerID="8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae" exitCode=0
Oct 31 14:47:19.830557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.830381    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db59ccdd5-29tbc" event={"ID":"5a91f700-90f1-4e51-94d9-d7d2320fc16e","Type":"ContainerDied","Data":"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"}
Oct 31 14:47:19.830557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.830401    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db59ccdd5-29tbc"
Oct 31 14:47:19.830557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.830459    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db59ccdd5-29tbc" event={"ID":"5a91f700-90f1-4e51-94d9-d7d2320fc16e","Type":"ContainerDied","Data":"ad0af696d578def7dfb33a8dc6acada1ebaa486844c88e3a6cf1b7f00f4cc68d"}
Oct 31 14:47:19.830557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.830484    4549 scope.go:122] "RemoveContainer" containerID="8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"
Oct 31 14:47:19.838244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.838230    4549 scope.go:122] "RemoveContainer" containerID="8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"
Oct 31 14:47:19.838499 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:19.838480    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae\": container with ID starting with 8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae not found: ID does not exist" containerID="8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"
Oct 31 14:47:19.838549 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.838511    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae"} err="failed to get container status \"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae\": rpc error: code = NotFound desc = could not find container \"8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae\": container with ID starting with 8b7fc5849b2d260ad9eded83d4394c6c9c3ce5b0a20ddb9ef585ae358b0498ae not found: ID does not exist"
Oct 31 14:47:19.992322 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:19.992288    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:47:20.041870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:20.041837    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6db59ccdd5-29tbc"]
Oct 31 14:47:21.416944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:21.416915    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" path="/var/lib/kubelet/pods/5a91f700-90f1-4e51-94d9-d7d2320fc16e/volumes"
Oct 31 14:47:22.079921 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.079891    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:22.080197 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080175    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="alertmanager" containerID="cri-o://cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06" gracePeriod=120
Oct 31 14:47:22.080285 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080252    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-metric" containerID="cri-o://728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69" gracePeriod=120
Oct 31 14:47:22.080450 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080283    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-web" containerID="cri-o://19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74" gracePeriod=120
Oct 31 14:47:22.080450 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080296    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy" containerID="cri-o://6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8" gracePeriod=120
Oct 31 14:47:22.080450 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080327    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="config-reloader" containerID="cri-o://ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4" gracePeriod=120
Oct 31 14:47:22.080450 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.080308    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="prom-label-proxy" containerID="cri-o://03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df" gracePeriod=120
Oct 31 14:47:22.847151 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847112    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df" exitCode=0
Oct 31 14:47:22.847151 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847141    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69" exitCode=0
Oct 31 14:47:22.847151 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847148    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8" exitCode=0
Oct 31 14:47:22.847151 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847155    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4" exitCode=0
Oct 31 14:47:22.847151 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847145    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"}
Oct 31 14:47:22.847754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847160    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06" exitCode=0
Oct 31 14:47:22.847754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847233    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"}
Oct 31 14:47:22.847754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847278    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"}
Oct 31 14:47:22.847754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847322    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"}
Oct 31 14:47:22.847754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:22.847358    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"}
Oct 31 14:47:23.317902 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.317880    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:23.422074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422009    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422042    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422064    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422304 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422081    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422304 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422109    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422304 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422253    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422302    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422345    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422389    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422423    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422447    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422486    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.422743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422503    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:23.422743 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422560    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\") pod \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\" (UID: \"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095\") "
Oct 31 14:47:23.423139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.422964    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-metrics-client-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.423780 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.423171    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:23.423780 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.423748    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:23.425786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.425733    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:23.425878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.425843    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.426068 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.426030    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.426231 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.426172    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.426352 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.426332    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:23.426427 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.426395    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.426653 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.426629    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.427216 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.427186    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "kube-api-access-2dss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:23.429686 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.429546    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.436063 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.436042    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config" pod "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" (UID: "1c951f1d-c2d6-49f3-80e1-d9b1be5e8095"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:23.523406 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523375    4549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523406 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523399    4549 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-out\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523406 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523408    4549 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-main-db\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523406 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523417    4549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523428    4549 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-tls-assets\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523437    4549 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-web-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523445    4549 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523455    4549 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523464    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dss7\" (UniqueName: \"kubernetes.io/projected/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-kube-api-access-2dss7\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523472    4549 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-cluster-tls-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523481    4549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.523695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.523489    4549 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095-secret-alertmanager-main-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:23.853679 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.853643    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerID="19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74" exitCode=0
Oct 31 14:47:23.854079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.853724    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"}
Oct 31 14:47:23.854079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.853749    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:23.854079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.853804    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c951f1d-c2d6-49f3-80e1-d9b1be5e8095","Type":"ContainerDied","Data":"77c1156d518f8225fce00323e891877f070b359d5c903438f7e0ee4db533f493"}
Oct 31 14:47:23.854079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.853838    4549 scope.go:122] "RemoveContainer" containerID="03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"
Oct 31 14:47:23.861825 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.861794    4549 scope.go:122] "RemoveContainer" containerID="728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"
Oct 31 14:47:23.868749 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.868730    4549 scope.go:122] "RemoveContainer" containerID="6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"
Oct 31 14:47:23.875412 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.875396    4549 scope.go:122] "RemoveContainer" containerID="19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"
Oct 31 14:47:23.882165 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.882146    4549 scope.go:122] "RemoveContainer" containerID="ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"
Oct 31 14:47:23.889111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.889093    4549 scope.go:122] "RemoveContainer" containerID="cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"
Oct 31 14:47:23.895919 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.895895    4549 scope.go:122] "RemoveContainer" containerID="d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709"
Oct 31 14:47:23.902253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.902232    4549 scope.go:122] "RemoveContainer" containerID="03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"
Oct 31 14:47:23.902506 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.902490    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df\": container with ID starting with 03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df not found: ID does not exist" containerID="03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"
Oct 31 14:47:23.902562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.902517    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df"} err="failed to get container status \"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df\": rpc error: code = NotFound desc = could not find container \"03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df\": container with ID starting with 03c2bd4b7c8db69e5c53e80be82cf4ee1e7da8a75b660971f0c8a2c2466ba6df not found: ID does not exist"
Oct 31 14:47:23.902562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.902541    4549 scope.go:122] "RemoveContainer" containerID="728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"
Oct 31 14:47:23.902846 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.902828    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69\": container with ID starting with 728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69 not found: ID does not exist" containerID="728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"
Oct 31 14:47:23.902896 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.902853    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69"} err="failed to get container status \"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69\": rpc error: code = NotFound desc = could not find container \"728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69\": container with ID starting with 728a3e65d2f7541c94c49ce42f820d73e0d3806384a080fb3832d5d6cdd73b69 not found: ID does not exist"
Oct 31 14:47:23.902896 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.902869    4549 scope.go:122] "RemoveContainer" containerID="6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"
Oct 31 14:47:23.903095 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.903080    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8\": container with ID starting with 6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8 not found: ID does not exist" containerID="6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"
Oct 31 14:47:23.903140 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903101    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8"} err="failed to get container status \"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8\": rpc error: code = NotFound desc = could not find container \"6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8\": container with ID starting with 6fd601aa306afc196cd5c244755a0cd94a7ba39983c40e6dd71ab97dc2c17dd8 not found: ID does not exist"
Oct 31 14:47:23.903140 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903114    4549 scope.go:122] "RemoveContainer" containerID="19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"
Oct 31 14:47:23.903368 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.903349    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74\": container with ID starting with 19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74 not found: ID does not exist" containerID="19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"
Oct 31 14:47:23.903410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903371    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74"} err="failed to get container status \"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74\": rpc error: code = NotFound desc = could not find container \"19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74\": container with ID starting with 19fe5180b9ec12bdee8ad88bec1a422bfb98430c3a4eb24fb304ca1681cf9c74 not found: ID does not exist"
Oct 31 14:47:23.903410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903385    4549 scope.go:122] "RemoveContainer" containerID="ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"
Oct 31 14:47:23.903644 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.903627    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4\": container with ID starting with ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4 not found: ID does not exist" containerID="ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"
Oct 31 14:47:23.903689 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903649    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4"} err="failed to get container status \"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4\": rpc error: code = NotFound desc = could not find container \"ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4\": container with ID starting with ebb130a300a496aa7c7421d4f8307f38e8a2133c90f476e7ef3aba631a36c8b4 not found: ID does not exist"
Oct 31 14:47:23.903689 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903662    4549 scope.go:122] "RemoveContainer" containerID="cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"
Oct 31 14:47:23.903891 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.903875    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06\": container with ID starting with cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06 not found: ID does not exist" containerID="cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"
Oct 31 14:47:23.903930 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903897    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06"} err="failed to get container status \"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06\": rpc error: code = NotFound desc = could not find container \"cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06\": container with ID starting with cc23382a703aa030f092eb338cce28143f569fc347dd27b08a32d683aa816b06 not found: ID does not exist"
Oct 31 14:47:23.903930 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.903911    4549 scope.go:122] "RemoveContainer" containerID="d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709"
Oct 31 14:47:23.904142 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:23.904125    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709\": container with ID starting with d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709 not found: ID does not exist" containerID="d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709"
Oct 31 14:47:23.904199 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:23.904151    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709"} err="failed to get container status \"d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709\": rpc error: code = NotFound desc = could not find container \"d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709\": container with ID starting with d62cc5a3a17aadc066b4d19d10c240200b845841bdf1e85bdae431738705e709 not found: ID does not exist"
Oct 31 14:47:24.019581 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.019535    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:24.069766 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.069728    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:24.070442 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070419    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:24.070740 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070725    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="init-config-reloader"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070742    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="init-config-reloader"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070758    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" containerName="console"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070765    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" containerName="console"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070776    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070784    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070805    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="alertmanager"
Oct 31 14:47:24.070821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070812    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="alertmanager"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070824    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-web"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070831    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-web"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070846    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="config-reloader"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070854    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="config-reloader"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070867    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-metric"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070874    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-metric"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070886    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="prom-label-proxy"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070893    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="prom-label-proxy"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070964    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="alertmanager"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070975    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-web"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070986    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="config-reloader"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.070994    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy-metric"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.071002    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="prom-label-proxy"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.071013    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a91f700-90f1-4e51-94d9-d7d2320fc16e" containerName="console"
Oct 31 14:47:24.071152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.071020    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" containerName="kube-rbac-proxy"
Oct 31 14:47:24.076167 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.076148    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128615    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128652    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128679    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128814    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128844    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.128955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128921    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129072 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.128970    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-web-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129072 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129023    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129072 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129061    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-out\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129076    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-volume\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129094    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129187    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129298 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129246    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pq8k\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-kube-api-access-4pq8k\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.129534 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129510    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 14:47:24.129534 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129510    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 14:47:24.129770 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129725    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 14:47:24.129770 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129752    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 14:47:24.129770 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129769    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 14:47:24.129954 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129784    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 14:47:24.129954 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129800    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 14:47:24.130044 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.129964    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 14:47:24.130044 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.130035    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gfp4g\""
Oct 31 14:47:24.179079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.179058    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 14:47:24.185622 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.185596    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:24.229897 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.229862    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.229897 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.229898    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.229920    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.229947    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.229976    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-web-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230005    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-out\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230063    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-volume\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230086    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230115    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230661 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230137    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pq8k\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-kube-api-access-4pq8k\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230661 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230164    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230661 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230193    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.230977 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.230753    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.231841 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.231816    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.232402 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.232374    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c848632e-78b5-405a-ba1a-b7b22fac7a91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.233303 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.233265    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-out\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.233397 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.233365    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.233520 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.233468    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.233646 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.233630    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.234188 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.234164    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-web-config\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.234363 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.234347    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.234511 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.234490    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.234621 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.234605    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-config-volume\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.235156 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.235141    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c848632e-78b5-405a-ba1a-b7b22fac7a91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.287432 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.287404    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pq8k\" (UniqueName: \"kubernetes.io/projected/c848632e-78b5-405a-ba1a-b7b22fac7a91-kube-api-access-4pq8k\") pod \"alertmanager-main-0\" (UID: \"c848632e-78b5-405a-ba1a-b7b22fac7a91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.400170 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.400090    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 14:47:24.825488 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.825463    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 14:47:24.826800 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:47:24.826770    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc848632e_78b5_405a_ba1a_b7b22fac7a91.slice/crio-8386aa5346fc303e67c98ce1872335c010f0bfb8f9fd45e72dc30222836d891d WatchSource:0}: Error finding container 8386aa5346fc303e67c98ce1872335c010f0bfb8f9fd45e72dc30222836d891d: Status 404 returned error can't find the container with id 8386aa5346fc303e67c98ce1872335c010f0bfb8f9fd45e72dc30222836d891d
Oct 31 14:47:24.859021 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:24.858991    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"8386aa5346fc303e67c98ce1872335c010f0bfb8f9fd45e72dc30222836d891d"}
Oct 31 14:47:25.419882 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:25.417812    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c951f1d-c2d6-49f3-80e1-d9b1be5e8095" path="/var/lib/kubelet/pods/1c951f1d-c2d6-49f3-80e1-d9b1be5e8095/volumes"
Oct 31 14:47:25.865523 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:25.865486    4549 generic.go:356] "Generic (PLEG): container finished" podID="c848632e-78b5-405a-ba1a-b7b22fac7a91" containerID="4a4402e1fa7652784e6a7061eaae4eb8864524e5935e63e7148aecc49dd8713e" exitCode=0
Oct 31 14:47:25.865902 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:25.865590    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerDied","Data":"4a4402e1fa7652784e6a7061eaae4eb8864524e5935e63e7148aecc49dd8713e"}
Oct 31 14:47:26.873999 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.873959    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"3cfd71e7ce18be91e030531d70a941122b633639b31c75e8846b2cc632dc74cb"}
Oct 31 14:47:26.874453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.874044    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"cd422daec92e5a83b02d3e42f90c004bd44365fd3e936aee82194d17b6c5a70e"}
Oct 31 14:47:26.874453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.874082    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"8330083fa8b3dda30b14de2a0449e8a1c3e1640d832cb4a3dc32292415915524"}
Oct 31 14:47:26.874453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.874118    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"493b82b427e2838de1a0d78621210af54b3d6ef876b1985dbffa2f428fd48203"}
Oct 31 14:47:26.874453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.874151    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"68bdf0f7f9b229671eeb29ae4c205cb9226016c5d26baaaeb617d3f3a3b18ec0"}
Oct 31 14:47:26.874453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.874184    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c848632e-78b5-405a-ba1a-b7b22fac7a91","Type":"ContainerStarted","Data":"e59e913fd11d641e6e98f02cd4d3f245216ebdd8c0fb3ce9f5d1017bf48969c7"}
Oct 31 14:47:26.998673 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:26.998613    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.986990666 podStartE2EDuration="2.998597267s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.606601ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:47:26.996874244 +0000 UTC m=+87.723155252" watchObservedRunningTime="2026-10-31 14:47:26.998597267 +0000 UTC m=+87.724878244"
Oct 31 14:47:28.243286 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:28.243248    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:28.243741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:28.243314    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:28.247909 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:28.247887    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:28.886949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:28.886924    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75b695586c-5g7g5"
Oct 31 14:47:29.050921 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:29.050894    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:47:37.010135 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:37.010096    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:47:54.070262 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.070217    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-666fbc7699-chgcd" podUID="73e64d5d-20c3-498c-9de4-227282d4df67" containerName="console" containerID="cri-o://a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672" gracePeriod=15
Oct 31 14:47:54.314115 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.314091    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:47:54.484367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484279    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484320    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484377    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484410    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484442    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484493    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\" (UniqueName: \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484524    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484554    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\") pod \"73e64d5d-20c3-498c-9de4-227282d4df67\" (UID: \"73e64d5d-20c3-498c-9de4-227282d4df67\") "
Oct 31 14:47:54.484919 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484804    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:47:54.484919 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484858    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:54.485030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484953    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:54.485199 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.484976    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:54.485312 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.485138    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:47:54.486738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.486715    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:54.486833 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.486815    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "kube-api-access-4jd6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:47:54.486883 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.486859    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert" pod "73e64d5d-20c3-498c-9de4-227282d4df67" (UID: "73e64d5d-20c3-498c-9de4-227282d4df67"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:47:54.585492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585443    4549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585485    4549 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73e64d5d-20c3-498c-9de4-227282d4df67-tmp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585497    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jd6d\" (UniqueName: \"kubernetes.io/projected/73e64d5d-20c3-498c-9de4-227282d4df67-kube-api-access-4jd6d\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585505    4549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-oauth-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585514    4549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-serving-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585522    4549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73e64d5d-20c3-498c-9de4-227282d4df67-console-oauth-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585530    4549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-console-config\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.585888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.585538    4549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73e64d5d-20c3-498c-9de4-227282d4df67-trusted-ca-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:47:54.982926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.982895    4549 generic.go:356] "Generic (PLEG): container finished" podID="73e64d5d-20c3-498c-9de4-227282d4df67" containerID="a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672" exitCode=0
Oct 31 14:47:54.983092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.982932    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fbc7699-chgcd" event={"ID":"73e64d5d-20c3-498c-9de4-227282d4df67","Type":"ContainerDied","Data":"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"}
Oct 31 14:47:54.983092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.982940    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fbc7699-chgcd"
Oct 31 14:47:54.983092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.982973    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fbc7699-chgcd" event={"ID":"73e64d5d-20c3-498c-9de4-227282d4df67","Type":"ContainerDied","Data":"82da1b0d0c0b419459cd4e1a28ab66a04c6ef68fdb5f2556afaf2220880a8668"}
Oct 31 14:47:54.983092 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.982992    4549 scope.go:122] "RemoveContainer" containerID="a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"
Oct 31 14:47:54.991473 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.991450    4549 scope.go:122] "RemoveContainer" containerID="a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"
Oct 31 14:47:54.991791 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:47:54.991773    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672\": container with ID starting with a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672 not found: ID does not exist" containerID="a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"
Oct 31 14:47:54.991852 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:54.991800    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672"} err="failed to get container status \"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672\": rpc error: code = NotFound desc = could not find container \"a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672\": container with ID starting with a0aac24e660255ecbd41b6bf04eabb0d9e91eacbf9441fbd65ac2c4cda13b672 not found: ID does not exist"
Oct 31 14:47:55.145192 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:55.145160    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:47:55.194847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:55.194813    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666fbc7699-chgcd"]
Oct 31 14:47:55.416617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:47:55.416592    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e64d5d-20c3-498c-9de4-227282d4df67" path="/var/lib/kubelet/pods/73e64d5d-20c3-498c-9de4-227282d4df67/volumes"
Oct 31 14:48:07.009973 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:48:07.009938    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:48:14.037278 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.037198    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"]
Oct 31 14:48:14.037659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.037472    4549 cpu_manager.go:414] "containerMap: removing container" podUID="73e64d5d-20c3-498c-9de4-227282d4df67" containerName="console"
Oct 31 14:48:14.037659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.037482    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e64d5d-20c3-498c-9de4-227282d4df67" containerName="console"
Oct 31 14:48:14.037659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.037547    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e64d5d-20c3-498c-9de4-227282d4df67" containerName="console"
Oct 31 14:48:14.040465 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.040449    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.128876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.128842    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.128876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.128881    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.129072 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.128905    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszcr\" (UniqueName: \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.144089 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.144065    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4qff8\""
Oct 31 14:48:14.144247 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.144123    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 14:48:14.144309 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.144266    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 14:48:14.145218 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.145196    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"]
Oct 31 14:48:14.145928 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.145912    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"]
Oct 31 14:48:14.149127 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.149111    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.204587 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.204554    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-gkp5n\""
Oct 31 14:48:14.208941 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.208916    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"]
Oct 31 14:48:14.229769 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.229745    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.230059 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.229774    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.230059 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.229793    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszcr\" (UniqueName: \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.230059 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.229992    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrqs\" (UniqueName: \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.230198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.230131    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.230198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.230160    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.230198 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.230177    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.230290 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.230201    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.238419 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.238398    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"]
Oct 31 14:48:14.241690 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.241675    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.287127 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.287094    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszcr\" (UniqueName: \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.299103 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.299052    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"]
Oct 31 14:48:14.304654 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.304630    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ptdlx\""
Oct 31 14:48:14.326969 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.326942    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"]
Oct 31 14:48:14.330365 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330345    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.330451 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330404    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"]
Oct 31 14:48:14.330700 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330661    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrqs\" (UniqueName: \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.330822 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330716    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.330822 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330742    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.330951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330905    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.330951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330945    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.331071 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.330979    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtsn\" (UniqueName: \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.331306 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.331286    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.331350 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.331294    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.333741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.333727    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.355814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.355788    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:48:14.387760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.387725    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"]
Oct 31 14:48:14.390114 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.390065    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrqs\" (UniqueName: \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.400272 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.400244    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"]
Oct 31 14:48:14.405135 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.405115    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ddl84\""
Oct 31 14:48:14.405258 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.405220    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-npk2m\""
Oct 31 14:48:14.431933 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.431908    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.432067 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.431943    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.432067 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.431973    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.432067 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432045    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnht\" (UniqueName: \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432128    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432186    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432232    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432238    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj4h\" (UniqueName: \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432302    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432317    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtsn\" (UniqueName: \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.432607 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.432382    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.464506 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.464478    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:48:14.489640 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.489615    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtsn\" (UniqueName: \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.533019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.532994    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.533174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533033    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.533174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533064    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnht\" (UniqueName: \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.533174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533091    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.533174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533131    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.533174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533174    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj4h\" (UniqueName: \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.533475 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533453    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.533548 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533472    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.533659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533543    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.533728 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.533675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.559919 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.559862    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:48:14.589863 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.589838    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnht\" (UniqueName: \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.590108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.590082    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj4h\" (UniqueName: \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.647158 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.647131    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:48:14.653170 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.653149    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:48:14.778367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.778340    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"]
Oct 31 14:48:14.781026 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:14.781001    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35f6a84_76bb_452f_9e98_6fb4225bb39d.slice/crio-759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe WatchSource:0}: Error finding container 759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe: Status 404 returned error can't find the container with id 759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe
Oct 31 14:48:14.832646 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.832538    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"]
Oct 31 14:48:14.835157 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:14.835133    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf660270_8eae_4c52_9d86_de3c0e0dfde3.slice/crio-91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22 WatchSource:0}: Error finding container 91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22: Status 404 returned error can't find the container with id 91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22
Oct 31 14:48:14.927252 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:14.927122    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"]
Oct 31 14:48:14.929753 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:14.929724    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc1e8f3_40ed_4031_9ef2_139114477860.slice/crio-d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8 WatchSource:0}: Error finding container d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8: Status 404 returned error can't find the container with id d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8
Oct 31 14:48:15.015793 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.015759    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"]
Oct 31 14:48:15.021059 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:15.021028    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abe7241_88d3_4dc4_bb8b_1853b4a5fc87.slice/crio-53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832 WatchSource:0}: Error finding container 53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832: Status 404 returned error can't find the container with id 53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832
Oct 31 14:48:15.022244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.022222    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"]
Oct 31 14:48:15.025169 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:15.025147    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e0435_521b_4c35_bd44_45e57da5936e.slice/crio-352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255 WatchSource:0}: Error finding container 352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255: Status 404 returned error can't find the container with id 352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255
Oct 31 14:48:15.058418 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.058393    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr" event={"ID":"bd2e0435-521b-4c35-bd44-45e57da5936e","Type":"ContainerStarted","Data":"352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255"}
Oct 31 14:48:15.059453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.059431    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz" event={"ID":"1dc1e8f3-40ed-4031-9ef2-139114477860","Type":"ContainerStarted","Data":"d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8"}
Oct 31 14:48:15.060593 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.060556    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml" event={"ID":"af660270-8eae-4c52-9d86-de3c0e0dfde3","Type":"ContainerStarted","Data":"91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22"}
Oct 31 14:48:15.061616 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.061596    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k" event={"ID":"b35f6a84-76bb-452f-9e98-6fb4225bb39d","Type":"ContainerStarted","Data":"759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe"}
Oct 31 14:48:15.062623 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.062602    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv" event={"ID":"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87","Type":"ContainerStarted","Data":"53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832"}
Oct 31 14:48:15.940611 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.938884    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"]
Oct 31 14:48:15.945584 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:15.945545    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.000542 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.000508    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"]
Oct 31 14:48:16.047942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.047882    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-pw8ft\""
Oct 31 14:48:16.049847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.049592    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.049847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.049649    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.049847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.049690    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfvs\" (UniqueName: \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.150889 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.150860    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.151362 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.150903    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.151362 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.150936    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfvs\" (UniqueName: \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.151685 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.151660    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.151932 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.151913    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.212786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.212684    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfvs\" (UniqueName: \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.269209 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.268956    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:48:16.667221 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:16.667186    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"]
Oct 31 14:48:17.084164 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:17.084069    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2" event={"ID":"21203dac-8909-4a4d-ab9e-2fec1589db0b","Type":"ContainerStarted","Data":"60baa46e26dd9c254f14461672370a026c4a717cbe6dc8ddc4b9dc9971196687"}
Oct 31 14:48:18.243775 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.243738    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"]
Oct 31 14:48:18.254771 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.254742    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.305614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.305520    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"]
Oct 31 14:48:18.307241 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.307164    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n42mj\""
Oct 31 14:48:18.373827 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.373582    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.373827 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.373650    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpk6\" (UniqueName: \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.373827 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.373696    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.475358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.474264    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.475358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.474343    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpk6\" (UniqueName: \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.475358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.474387    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.475358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.474852    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.475358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.475074    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.545130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.545066    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpk6\" (UniqueName: \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:18.577953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:18.577918    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:48:19.010894 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.010860    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"]
Oct 31 14:48:19.103876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.103794    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerStarted","Data":"b67d1707c4607efb2747a419a016f56e7a74cff8e3cd01dd43640dd76ac4f62c"}
Oct 31 14:48:19.888108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.888075    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 14:48:19.893162 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.893133    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.946616 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.946547    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\""
Oct 31 14:48:19.972265 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.972126    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 14:48:19.972265 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.972203    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987044    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987111    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987142    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987171    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8n6j\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987200    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987249    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987325    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987346    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.987675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.987372    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:19.998924 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.998858    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 14:48:20.000009 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:19.999787    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088011    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088089    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088117    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088146    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088203    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088242    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088273    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088303    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8n6j\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.088332    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.089095    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.090050 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.089773    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.092150 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.090987    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.092150 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.091504    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.092150 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.091765    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.098930 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.098909    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.101880 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.101815    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.155003 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.154929    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.159153 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.159104    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8n6j\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\") pod \"maas-default-gateway-openshift-default-67bf986f99-x6jcg\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.220664 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.220629    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:20.226271 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.226241    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"]
Oct 31 14:48:20.230423 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.230151    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.286012 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.283907    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-76klk\""
Oct 31 14:48:20.293766 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.293675    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"]
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302234    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0774f12c-928b-4705-b474-3a0afa129dcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302331    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302416    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302452    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302494    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302615    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302731    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fsk\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-kube-api-access-q7fsk\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302809    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0774f12c-928b-4705-b474-3a0afa129dcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.303169 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.302851    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.403993    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fsk\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-kube-api-access-q7fsk\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404040    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0774f12c-928b-4705-b474-3a0afa129dcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404079    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404124    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0774f12c-928b-4705-b474-3a0afa129dcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404184    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404207    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404240    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404282    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.404304    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.405026    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.405979 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.405900    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0774f12c-928b-4705-b474-3a0afa129dcf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.406133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.406032    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.406231 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.406168    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.406775 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.406718    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.413215 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.413168    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0774f12c-928b-4705-b474-3a0afa129dcf-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.417272 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.417245    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0774f12c-928b-4705-b474-3a0afa129dcf-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.469584 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.469515    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.473455 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.473352    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fsk\" (UniqueName: \"kubernetes.io/projected/0774f12c-928b-4705-b474-3a0afa129dcf-kube-api-access-q7fsk\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8\" (UID: \"0774f12c-928b-4705-b474-3a0afa129dcf\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.564441 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.564410    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:20.703992 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.702848    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 14:48:20.707848 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:20.707809    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3375e0e0_48b4_4232_95af_66e6751482f0.slice/crio-8633b7100da6c4d5250461428cdc41556c7003113adb2702f1984ef0871590f5 WatchSource:0}: Error finding container 8633b7100da6c4d5250461428cdc41556c7003113adb2702f1984ef0871590f5: Status 404 returned error can't find the container with id 8633b7100da6c4d5250461428cdc41556c7003113adb2702f1984ef0871590f5
Oct 31 14:48:20.882235 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.880191    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"]
Oct 31 14:48:20.886175 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.885558    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:20.904130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.904101    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"]
Oct 31 14:48:20.909864 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.909498    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:20.935496 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.933968    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"]
Oct 31 14:48:20.937657 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.937640    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:20.946290 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.946257    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"]
Oct 31 14:48:20.963616 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.963528    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"]
Oct 31 14:48:20.969007 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.968982    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:20.973244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:20.973217    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"]
Oct 31 14:48:21.004925 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.004890    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"]
Oct 31 14:48:21.010989 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.010957    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.011145 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011016    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.011145 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011074    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.011145 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011113    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.011321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011186    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.011321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011246    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs4f\" (UniqueName: \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.011321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011262    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.011321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011282    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7nk\" (UniqueName: \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.011321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011323    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.011608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011352    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.011608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011377    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6stq\" (UniqueName: \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.011608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.011405    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.024886 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.024773    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"]
Oct 31 14:48:21.029703 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.029628    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"]
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112384    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112430    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs4f\" (UniqueName: \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112457    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112480    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7nk\" (UniqueName: \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112508    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112530    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112552    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6stq\" (UniqueName: \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112586    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112610    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112635    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112648    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.112684    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.113327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113023    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113396    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113559    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113741    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113943    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.113971    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.114142 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.114129    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.114947 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.114892    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.126443 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.126405    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8" event={"ID":"0774f12c-928b-4705-b474-3a0afa129dcf","Type":"ContainerStarted","Data":"cb0b6574a943f8839110784a518ea1b974fe424f535432c97f25dfca397d154d"}
Oct 31 14:48:21.135957 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.135861    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerStarted","Data":"8633b7100da6c4d5250461428cdc41556c7003113adb2702f1984ef0871590f5"}
Oct 31 14:48:21.175680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.175016    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.175680 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.175632    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7nk\" (UniqueName: \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.176999 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.176948    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6stq\" (UniqueName: \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.181086 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.181029    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs4f\" (UniqueName: \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.207532 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.207506    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:48:21.229056 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.229025    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:48:21.260212 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.258895    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:48:21.290755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.290723    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:48:21.615868 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.615778    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"]
Oct 31 14:48:21.631563 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:48:21.631514    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46e3d6a_66d0_44ba_8628_cd7dbd7ea633.slice/crio-d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5 WatchSource:0}: Error finding container d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5: Status 404 returned error can't find the container with id d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5
Oct 31 14:48:21.698730 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.697157    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"]
Oct 31 14:48:21.698730 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.697217    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"]
Oct 31 14:48:21.702502 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:21.702481    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"]
Oct 31 14:48:22.162526 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:22.162487    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerStarted","Data":"a48ab784123849f42322470bfa70f5c3e8dd3687c2af678674ca4350c8b8f515"}
Oct 31 14:48:22.169786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:22.169688    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerStarted","Data":"d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5"}
Oct 31 14:48:22.193881 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:22.193792    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerStarted","Data":"7f5c35c5bc750bb43705cf43396a94ee80017ce49a1cbddc318544c8d1f67969"}
Oct 31 14:48:22.200366 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:22.200336    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7" event={"ID":"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82","Type":"ContainerStarted","Data":"50934f11d7a4b146a84b91ec0f5561bac83671fee2196be507407b623c592407"}
Oct 31 14:48:34.072492 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.072452    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.072952 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.072551    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.073027 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.072987    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.079882 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.079795    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.079959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.079876    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.079959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.079922    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:48:34.373558 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.372677    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8" event={"ID":"0774f12c-928b-4705-b474-3a0afa129dcf","Type":"ContainerStarted","Data":"cbe045f87ba197103093eff22168508700572c1be666559277b6eee494cc3a86"}
Oct 31 14:48:34.380402 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.379797    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerStarted","Data":"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"}
Oct 31 14:48:34.508111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.508053    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8" podStartSLOduration=1.478864955 podStartE2EDuration="14.508035653s" totalImagesPullingTime="13.029170698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:48:34.504447383 +0000 UTC m=+155.230728362" watchObservedRunningTime="2026-10-31 14:48:34.508035653 +0000 UTC m=+155.234316632"
Oct 31 14:48:34.564988 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:34.564944    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:35.223425 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.221660    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:35.224139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.223994    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:35.228985 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.228962    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:35.348240 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.348178    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podStartSLOduration=2.980429043 podStartE2EDuration="16.348156044s" totalImagesPullingTime="13.367727001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:48:34.631338328 +0000 UTC m=+155.357619309" watchObservedRunningTime="2026-10-31 14:48:35.348156044 +0000 UTC m=+156.074437023"
Oct 31 14:48:35.390464 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.390251    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:48:35.565408 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.565372    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:35.571848 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:35.571820    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:36.393864 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:36.393831    4549 generic.go:356] "Generic (PLEG): container finished" podID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerID="e8a70c781dc062742afcba80fe45f8b8d07482573be4c1a244295704381d5dc5" exitCode=0
Oct 31 14:48:36.394341 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:36.393918    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml" event={"ID":"af660270-8eae-4c52-9d86-de3c0e0dfde3","Type":"ContainerDied","Data":"e8a70c781dc062742afcba80fe45f8b8d07482573be4c1a244295704381d5dc5"}
Oct 31 14:48:36.396478 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:36.396438    4549 generic.go:356] "Generic (PLEG): container finished" podID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerID="4717664bc25710c4d1550a61a387e0f6abfbae934891bcb8114cb429372e3a4a" exitCode=0
Oct 31 14:48:36.396637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:36.396542    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k" event={"ID":"b35f6a84-76bb-452f-9e98-6fb4225bb39d","Type":"ContainerDied","Data":"4717664bc25710c4d1550a61a387e0f6abfbae934891bcb8114cb429372e3a4a"}
Oct 31 14:48:36.398130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:36.398110    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8"
Oct 31 14:48:37.019884 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:48:37.019847    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:48:37.403841 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.403755    4549 generic.go:356] "Generic (PLEG): container finished" podID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerID="5338582f77069534c3cb76f8ba3f9fef9397a00b34364927b2253907db14327e" exitCode=0
Oct 31 14:48:37.404496 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.404472    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz" event={"ID":"1dc1e8f3-40ed-4031-9ef2-139114477860","Type":"ContainerDied","Data":"5338582f77069534c3cb76f8ba3f9fef9397a00b34364927b2253907db14327e"}
Oct 31 14:48:37.406620 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.406595    4549 generic.go:356] "Generic (PLEG): container finished" podID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerID="a78a1a8598ad632958f743d51173d2815f438d314a4a6dbd3a4d3c816ce03954" exitCode=0
Oct 31 14:48:37.406734 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.406679    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerDied","Data":"a78a1a8598ad632958f743d51173d2815f438d314a4a6dbd3a4d3c816ce03954"}
Oct 31 14:48:37.408408 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.408383    4549 generic.go:356] "Generic (PLEG): container finished" podID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerID="7badc58ef8dedaad4b519cbea80cae5e7aa8576f7377a10ca0d59cee6dbb754e" exitCode=0
Oct 31 14:48:37.408549 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.408526    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7" event={"ID":"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82","Type":"ContainerDied","Data":"7badc58ef8dedaad4b519cbea80cae5e7aa8576f7377a10ca0d59cee6dbb754e"}
Oct 31 14:48:37.410888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.410339    4549 generic.go:356] "Generic (PLEG): container finished" podID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerID="2071e659c1cd268dbd91948a5ac3c1d8436b01b32d226b156f10ed039f75ada4" exitCode=0
Oct 31 14:48:37.410888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:37.410406    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv" event={"ID":"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87","Type":"ContainerDied","Data":"2071e659c1cd268dbd91948a5ac3c1d8436b01b32d226b156f10ed039f75ada4"}
Oct 31 14:48:38.418497 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.418465    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerID="2f1f100c84d39bca3f909bd1bc55930dd8de42747cd605e36f2fefcb3f066dbd" exitCode=0
Oct 31 14:48:38.418950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.418551    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr" event={"ID":"bd2e0435-521b-4c35-bd44-45e57da5936e","Type":"ContainerDied","Data":"2f1f100c84d39bca3f909bd1bc55930dd8de42747cd605e36f2fefcb3f066dbd"}
Oct 31 14:48:38.420565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.420481    4549 generic.go:356] "Generic (PLEG): container finished" podID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerID="02abada3733eccb7ee3af802f18925e81bd182059c98e8e7c47e4a1f66e885cd" exitCode=0
Oct 31 14:48:38.420666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.420590    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerDied","Data":"02abada3733eccb7ee3af802f18925e81bd182059c98e8e7c47e4a1f66e885cd"}
Oct 31 14:48:38.422613 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.422591    4549 generic.go:356] "Generic (PLEG): container finished" podID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerID="2346475ed6ee50087ae2c91d5b9f565da3c01c2c2ba00232ee6947867220fafa" exitCode=0
Oct 31 14:48:38.422694 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.422668    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerDied","Data":"2346475ed6ee50087ae2c91d5b9f565da3c01c2c2ba00232ee6947867220fafa"}
Oct 31 14:48:38.424482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.424312    4549 generic.go:356] "Generic (PLEG): container finished" podID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerID="a8f0ff2c4b01a57a8935621bf1d689c8c945dab8e3b8d1b257d371641700d93f" exitCode=0
Oct 31 14:48:38.424482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.424343    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2" event={"ID":"21203dac-8909-4a4d-ab9e-2fec1589db0b","Type":"ContainerDied","Data":"a8f0ff2c4b01a57a8935621bf1d689c8c945dab8e3b8d1b257d371641700d93f"}
Oct 31 14:48:38.426553 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.426509    4549 generic.go:356] "Generic (PLEG): container finished" podID="7c49fa35-99d4-4627-9e36-323184c01f59" containerID="84a55ba2206a219be1042cb7c85426f42be2198c34c647eb4c33e1367f5a9857" exitCode=0
Oct 31 14:48:38.426687 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:38.426594    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerDied","Data":"84a55ba2206a219be1042cb7c85426f42be2198c34c647eb4c33e1367f5a9857"}
Oct 31 14:48:39.432562 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.432536    4549 generic.go:356] "Generic (PLEG): container finished" podID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerID="de78d70b542ab9b7858a992c1934ebdcc941da6411d2b1b47eba007707af5073" exitCode=0
Oct 31 14:48:39.432954 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.432606    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7" event={"ID":"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82","Type":"ContainerDied","Data":"de78d70b542ab9b7858a992c1934ebdcc941da6411d2b1b47eba007707af5073"}
Oct 31 14:48:39.434495 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.434393    4549 generic.go:356] "Generic (PLEG): container finished" podID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerID="7c4672a5f5259ce0df32662252305a02214eaaa90bc6127f8477934b66b78054" exitCode=0
Oct 31 14:48:39.434495 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.434475    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k" event={"ID":"b35f6a84-76bb-452f-9e98-6fb4225bb39d","Type":"ContainerDied","Data":"7c4672a5f5259ce0df32662252305a02214eaaa90bc6127f8477934b66b78054"}
Oct 31 14:48:39.436674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.436589    4549 generic.go:356] "Generic (PLEG): container finished" podID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerID="f54697bff82fad303fb7d4259066779156c0098c5580bb59e51f258cda798a96" exitCode=0
Oct 31 14:48:39.436674 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.436656    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv" event={"ID":"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87","Type":"ContainerDied","Data":"f54697bff82fad303fb7d4259066779156c0098c5580bb59e51f258cda798a96"}
Oct 31 14:48:39.438800 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.438779    4549 generic.go:356] "Generic (PLEG): container finished" podID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerID="ee65694326e463173e89254373b0625486f9e92898d1912a1751d5a3f83891ea" exitCode=0
Oct 31 14:48:39.438887 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.438842    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz" event={"ID":"1dc1e8f3-40ed-4031-9ef2-139114477860","Type":"ContainerDied","Data":"ee65694326e463173e89254373b0625486f9e92898d1912a1751d5a3f83891ea"}
Oct 31 14:48:39.440684 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.440663    4549 generic.go:356] "Generic (PLEG): container finished" podID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerID="ace026bd7f228673074b1b523cd13683433d5c2d1023667eec62e300e21066e0" exitCode=0
Oct 31 14:48:39.440773 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.440740    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml" event={"ID":"af660270-8eae-4c52-9d86-de3c0e0dfde3","Type":"ContainerDied","Data":"ace026bd7f228673074b1b523cd13683433d5c2d1023667eec62e300e21066e0"}
Oct 31 14:48:39.442730 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.442711    4549 generic.go:356] "Generic (PLEG): container finished" podID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerID="f0a709cb2f5160108b568adf8e5c1bebc84fa4bd51436ec6eb9d92ac8dc599a2" exitCode=0
Oct 31 14:48:39.442824 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:39.442739    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerDied","Data":"f0a709cb2f5160108b568adf8e5c1bebc84fa4bd51436ec6eb9d92ac8dc599a2"}
Oct 31 14:48:40.469757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.469705    4549 generic.go:356] "Generic (PLEG): container finished" podID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerID="8f2317d0aad1465a5b022cb557f73b1bf5fcc42cec96d454c55bb1f0f9f304bf" exitCode=0
Oct 31 14:48:40.470353 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.469826    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2" event={"ID":"21203dac-8909-4a4d-ab9e-2fec1589db0b","Type":"ContainerDied","Data":"8f2317d0aad1465a5b022cb557f73b1bf5fcc42cec96d454c55bb1f0f9f304bf"}
Oct 31 14:48:40.482388 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.482356    4549 generic.go:356] "Generic (PLEG): container finished" podID="7c49fa35-99d4-4627-9e36-323184c01f59" containerID="5af5fc6e8632d20c2dd52b416e52a336037e4be51d127e9f0978d81014eb608a" exitCode=0
Oct 31 14:48:40.482564 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.482543    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerDied","Data":"5af5fc6e8632d20c2dd52b416e52a336037e4be51d127e9f0978d81014eb608a"}
Oct 31 14:48:40.493985 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.491041    4549 generic.go:356] "Generic (PLEG): container finished" podID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerID="ebfbea5a2fe2971571de8b1615627b134554404710b901d3cf031604ec4a4d5c" exitCode=0
Oct 31 14:48:40.493985 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:40.491171    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerDied","Data":"ebfbea5a2fe2971571de8b1615627b134554404710b901d3cf031604ec4a4d5c"}
Oct 31 14:48:45.595944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:45.594873    4549 generic.go:356] "Generic (PLEG): container finished" podID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerID="730736ad2062a1a5ef40c4a362b73928faa9e93ac053e0310bc6e1807a90904d" exitCode=0
Oct 31 14:48:45.595944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:45.594991    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerDied","Data":"730736ad2062a1a5ef40c4a362b73928faa9e93ac053e0310bc6e1807a90904d"}
Oct 31 14:48:47.683517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:47.683298    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerID="dcb911d575a51a3a24af68fb3a404ac6d710e5e306d8111a919d043557d6774c" exitCode=0
Oct 31 14:48:47.683517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:48:47.683387    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr" event={"ID":"bd2e0435-521b-4c35-bd44-45e57da5936e","Type":"ContainerDied","Data":"dcb911d575a51a3a24af68fb3a404ac6d710e5e306d8111a919d043557d6774c"}
Oct 31 14:49:02.951686 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:02.951629    4549 generic.go:356] "Generic (PLEG): container finished" podID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerID="eb7a5991469001ab0c0ead11c9a8679e9c5c10c8021d89b837436f01ec9f8750" exitCode=0
Oct 31 14:49:02.952234 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:02.951728    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz" event={"ID":"1dc1e8f3-40ed-4031-9ef2-139114477860","Type":"ContainerDied","Data":"eb7a5991469001ab0c0ead11c9a8679e9c5c10c8021d89b837436f01ec9f8750"}
Oct 31 14:49:02.955996 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:02.955968    4549 generic.go:356] "Generic (PLEG): container finished" podID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerID="151df8ea3dfb3f49c4cfd37193b8df67312e4251971aba74b171d5faffcbdab1" exitCode=0
Oct 31 14:49:02.956161 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:02.956060    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml" event={"ID":"af660270-8eae-4c52-9d86-de3c0e0dfde3","Type":"ContainerDied","Data":"151df8ea3dfb3f49c4cfd37193b8df67312e4251971aba74b171d5faffcbdab1"}
Oct 31 14:49:02.959617 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:02.959588    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerStarted","Data":"7dc145e533c66bed0651adddcf6292a79d334233718ad2bb450e697a0b0b2e0c"}
Oct 31 14:49:03.302445 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.302381    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" podStartSLOduration=3.241392893 podStartE2EDuration="43.302364959s" totalImagesPullingTime="40.060972066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:03.302069029 +0000 UTC m=+184.028350032" watchObservedRunningTime="2026-10-31 14:49:03.302364959 +0000 UTC m=+184.028645937"
Oct 31 14:49:03.968957 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.968929    4549 generic.go:356] "Generic (PLEG): container finished" podID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerID="cc6c3bb16a6ac2eb42cd074d96dd0a960df994ce200e6d4c36e634de148c949b" exitCode=0
Oct 31 14:49:03.969282 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.969031    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv" event={"ID":"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87","Type":"ContainerDied","Data":"cc6c3bb16a6ac2eb42cd074d96dd0a960df994ce200e6d4c36e634de148c949b"}
Oct 31 14:49:03.973210 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.972465    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerStarted","Data":"4fbd90571553747867042c008e47c9072e1613cd5193e69cf6cee67f26d6358e"}
Oct 31 14:49:03.975435 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.975412    4549 generic.go:356] "Generic (PLEG): container finished" podID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerID="7dc145e533c66bed0651adddcf6292a79d334233718ad2bb450e697a0b0b2e0c" exitCode=0
Oct 31 14:49:03.975529 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.975498    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerDied","Data":"7dc145e533c66bed0651adddcf6292a79d334233718ad2bb450e697a0b0b2e0c"}
Oct 31 14:49:03.977479 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.977459    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:49:03.978477 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.978453    4549 generic.go:356] "Generic (PLEG): container finished" podID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerID="0daad1cab9954b916e27aa897cb2e401b7a4754963520dc3d6e326ba80521712" exitCode=1
Oct 31 14:49:03.978633 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.978540    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7" event={"ID":"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82","Type":"ContainerDied","Data":"0daad1cab9954b916e27aa897cb2e401b7a4754963520dc3d6e326ba80521712"}
Oct 31 14:49:03.981482 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.981462    4549 generic.go:356] "Generic (PLEG): container finished" podID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerID="56841e00000a8790bca47dccf76a76b03a88ed7911a0362a9f7b64957b02d080" exitCode=0
Oct 31 14:49:03.981657 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.981637    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k" event={"ID":"b35f6a84-76bb-452f-9e98-6fb4225bb39d","Type":"ContainerDied","Data":"56841e00000a8790bca47dccf76a76b03a88ed7911a0362a9f7b64957b02d080"}
Oct 31 14:49:03.985145 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:03.985126    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerStarted","Data":"10d896c3553c2b38e93918438922f65263055a4f36a86ee78ff20b16852395ac"}
Oct 31 14:49:04.127625 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.127598    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:49:04.181610 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.181581    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:49:04.199219 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.199164    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" podStartSLOduration=3.348518772 podStartE2EDuration="44.199149449s" totalImagesPullingTime="40.850630677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:04.197871916 +0000 UTC m=+184.924152896" watchObservedRunningTime="2026-10-31 14:49:04.199149449 +0000 UTC m=+184.925430426"
Oct 31 14:49:04.223193 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.223159    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\" (UniqueName: \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\") pod \"1dc1e8f3-40ed-4031-9ef2-139114477860\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") "
Oct 31 14:49:04.223367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.223262    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\") pod \"1dc1e8f3-40ed-4031-9ef2-139114477860\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") "
Oct 31 14:49:04.223367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.223315    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\") pod \"1dc1e8f3-40ed-4031-9ef2-139114477860\" (UID: \"1dc1e8f3-40ed-4031-9ef2-139114477860\") "
Oct 31 14:49:04.224030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.223965    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle" pod "1dc1e8f3-40ed-4031-9ef2-139114477860" (UID: "1dc1e8f3-40ed-4031-9ef2-139114477860"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:04.226283 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.226231    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn" pod "1dc1e8f3-40ed-4031-9ef2-139114477860" (UID: "1dc1e8f3-40ed-4031-9ef2-139114477860"). InnerVolumeSpecName "kube-api-access-nvtsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:04.230971 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.230940    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util" pod "1dc1e8f3-40ed-4031-9ef2-139114477860" (UID: "1dc1e8f3-40ed-4031-9ef2-139114477860"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:04.324289 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324244    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\" (UniqueName: \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\") pod \"af660270-8eae-4c52-9d86-de3c0e0dfde3\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") "
Oct 31 14:49:04.324473 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324390    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\") pod \"af660270-8eae-4c52-9d86-de3c0e0dfde3\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") "
Oct 31 14:49:04.324473 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324426    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\") pod \"af660270-8eae-4c52-9d86-de3c0e0dfde3\" (UID: \"af660270-8eae-4c52-9d86-de3c0e0dfde3\") "
Oct 31 14:49:04.324667 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324650    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.324736 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324672    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1dc1e8f3-40ed-4031-9ef2-139114477860-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.324736 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.324686    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtsn\" (UniqueName: \"kubernetes.io/projected/1dc1e8f3-40ed-4031-9ef2-139114477860-kube-api-access-nvtsn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.325271 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.325247    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle" pod "af660270-8eae-4c52-9d86-de3c0e0dfde3" (UID: "af660270-8eae-4c52-9d86-de3c0e0dfde3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:04.333256 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.333220    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util" pod "af660270-8eae-4c52-9d86-de3c0e0dfde3" (UID: "af660270-8eae-4c52-9d86-de3c0e0dfde3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:04.335702 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.335671    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs" pod "af660270-8eae-4c52-9d86-de3c0e0dfde3" (UID: "af660270-8eae-4c52-9d86-de3c0e0dfde3"). InnerVolumeSpecName "kube-api-access-xrrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:04.425207 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.425173    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.425207 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.425207    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af660270-8eae-4c52-9d86-de3c0e0dfde3-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.425446 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.425227    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrqs\" (UniqueName: \"kubernetes.io/projected/af660270-8eae-4c52-9d86-de3c0e0dfde3-kube-api-access-xrrqs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:04.662093 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.662032    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" podStartSLOduration=4.367949871 podStartE2EDuration="44.662012392s" totalImagesPullingTime="40.294062521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:04.659843913 +0000 UTC m=+185.386124892" watchObservedRunningTime="2026-10-31 14:49:04.662012392 +0000 UTC m=+185.388293370"
Oct 31 14:49:04.993420 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.993338    4549 generic.go:356] "Generic (PLEG): container finished" podID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerID="90426d6668a967c6a9b53f4d8c1e8885953f50230fb4b57068a630262493a0a1" exitCode=0
Oct 31 14:49:04.993884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.993458    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2" event={"ID":"21203dac-8909-4a4d-ab9e-2fec1589db0b","Type":"ContainerDied","Data":"90426d6668a967c6a9b53f4d8c1e8885953f50230fb4b57068a630262493a0a1"}
Oct 31 14:49:04.996301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.996171    4549 generic.go:356] "Generic (PLEG): container finished" podID="7c49fa35-99d4-4627-9e36-323184c01f59" containerID="10d896c3553c2b38e93918438922f65263055a4f36a86ee78ff20b16852395ac" exitCode=0
Oct 31 14:49:04.996301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.996256    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerDied","Data":"10d896c3553c2b38e93918438922f65263055a4f36a86ee78ff20b16852395ac"}
Oct 31 14:49:04.999010 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.998906    4549 generic.go:356] "Generic (PLEG): container finished" podID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerID="4fbd90571553747867042c008e47c9072e1613cd5193e69cf6cee67f26d6358e" exitCode=0
Oct 31 14:49:04.999010 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:04.998987    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerDied","Data":"4fbd90571553747867042c008e47c9072e1613cd5193e69cf6cee67f26d6358e"}
Oct 31 14:49:05.001345 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.001091    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz" event={"ID":"1dc1e8f3-40ed-4031-9ef2-139114477860","Type":"ContainerDied","Data":"d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8"}
Oct 31 14:49:05.001345 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.001108    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7p9chz"
Oct 31 14:49:05.001345 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.001132    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5bce36af3612454000cb9c37e7a8e91e206412c76500311e1d7df72b1af73c8"
Oct 31 14:49:05.009708 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.009611    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml" event={"ID":"af660270-8eae-4c52-9d86-de3c0e0dfde3","Type":"ContainerDied","Data":"91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22"}
Oct 31 14:49:05.009708 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.009633    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvxnml"
Oct 31 14:49:05.009708 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.009660    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91bdca806e88e76618886edeebe91a3c10fa49459f006f1bb60b9e9d02cb3e22"
Oct 31 14:49:05.185168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.185070    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:49:05.221284 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.221255    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:49:05.267707 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.267683    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:49:05.269611 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.269590    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:49:05.271126 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.271106    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:49:05.335201 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335166    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\" (UniqueName: \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\") pod \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335225    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\") pod \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335254    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\" (UniqueName: \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\") pod \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335274    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\") pod \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335296    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\") pod \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\" (UID: \"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335313    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\") pod \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335329    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\" (UniqueName: \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\") pod \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335365    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\") pod \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\" (UID: \"b35f6a84-76bb-452f-9e98-6fb4225bb39d\") "
Oct 31 14:49:05.335399 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335397    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\") pod \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") "
Oct 31 14:49:05.335837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335415    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\") pod \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\" (UID: \"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82\") "
Oct 31 14:49:05.335837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335447    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\") pod \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") "
Oct 31 14:49:05.335837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.335489    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\") pod \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\" (UID: \"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6\") "
Oct 31 14:49:05.336613 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.336560    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle" pod "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" (UID: "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.336915 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.336885    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle" pod "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" (UID: "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.337380 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.337357    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle" pod "b35f6a84-76bb-452f-9e98-6fb4225bb39d" (UID: "b35f6a84-76bb-452f-9e98-6fb4225bb39d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.342546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.342504    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util" pod "b35f6a84-76bb-452f-9e98-6fb4225bb39d" (UID: "b35f6a84-76bb-452f-9e98-6fb4225bb39d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.342679 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.342598    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util" pod "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" (UID: "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.343178 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.343151    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle" pod "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" (UID: "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.343860 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.343834    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util" pod "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" (UID: "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.344655 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.344629    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util" pod "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" (UID: "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:05.347742 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.347710    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z" pod "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" (UID: "f44a9d6e-bf5a-4a9d-91b4-9733ed519a82"). InnerVolumeSpecName "kube-api-access-5lk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:05.347831 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.347740    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr" pod "b35f6a84-76bb-452f-9e98-6fb4225bb39d" (UID: "b35f6a84-76bb-452f-9e98-6fb4225bb39d"). InnerVolumeSpecName "kube-api-access-wszcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:05.348079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.347978    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f" pod "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" (UID: "6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6"). InnerVolumeSpecName "kube-api-access-nqs4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:05.348079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.348054    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h" pod "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" (UID: "3abe7241-88d3-4dc4-bb8b-1853b4a5fc87"). InnerVolumeSpecName "kube-api-access-slj4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436307    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436332    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slj4h\" (UniqueName: \"kubernetes.io/projected/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-kube-api-access-slj4h\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436341    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436350    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszcr\" (UniqueName: \"kubernetes.io/projected/b35f6a84-76bb-452f-9e98-6fb4225bb39d-kube-api-access-wszcr\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436358    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436366    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3abe7241-88d3-4dc4-bb8b-1853b4a5fc87-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436375    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk4z\" (UniqueName: \"kubernetes.io/projected/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-kube-api-access-5lk4z\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436383    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs4f\" (UniqueName: \"kubernetes.io/projected/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-kube-api-access-nqs4f\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436392    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f6a84-76bb-452f-9e98-6fb4225bb39d-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436404    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436413    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f44a9d6e-bf5a-4a9d-91b4-9733ed519a82-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:05.436469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:05.436420    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.016964 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.016927    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerStarted","Data":"38c53e311089b03a53cfa4ad7194938631905d06192e89a9f808560085afb459"}
Oct 31 14:49:06.018780 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.018750    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86" event={"ID":"6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6","Type":"ContainerDied","Data":"7f5c35c5bc750bb43705cf43396a94ee80017ce49a1cbddc318544c8d1f67969"}
Oct 31 14:49:06.018893 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.018793    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5c35c5bc750bb43705cf43396a94ee80017ce49a1cbddc318544c8d1f67969"
Oct 31 14:49:06.018893 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.018795    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7zkk86"
Oct 31 14:49:06.020248 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.020230    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:49:06.021084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.021069    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7"
Oct 31 14:49:06.021190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.021086    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7" event={"ID":"f44a9d6e-bf5a-4a9d-91b4-9733ed519a82","Type":"ContainerDied","Data":"50934f11d7a4b146a84b91ec0f5561bac83671fee2196be507407b623c592407"}
Oct 31 14:49:06.021190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.021127    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50934f11d7a4b146a84b91ec0f5561bac83671fee2196be507407b623c592407"
Oct 31 14:49:06.023017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.023000    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k"
Oct 31 14:49:06.023099 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.023022    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ld4k" event={"ID":"b35f6a84-76bb-452f-9e98-6fb4225bb39d","Type":"ContainerDied","Data":"759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe"}
Oct 31 14:49:06.023099 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.023056    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759a3dffb11233bd8c1e2b5a4f19d8ad62a7f7f91ea2689ef2febf6890764abe"
Oct 31 14:49:06.024798 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.024773    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv" event={"ID":"3abe7241-88d3-4dc4-bb8b-1853b4a5fc87","Type":"ContainerDied","Data":"53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832"}
Oct 31 14:49:06.024867 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.024811    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d3fd594bf1f52e8e79305be0dc9d230147b1f921ca0623016b0e1b64c57832"
Oct 31 14:49:06.025267 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.025229    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a2f9rv"
Oct 31 14:49:06.156010 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.155980    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:49:06.188678 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.188652    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:49:06.198869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.198843    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:49:06.244814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.244791    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\") pod \"21203dac-8909-4a4d-ab9e-2fec1589db0b\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") "
Oct 31 14:49:06.244990 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.244843    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\") pod \"21203dac-8909-4a4d-ab9e-2fec1589db0b\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") "
Oct 31 14:49:06.244990 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.244880    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\" (UniqueName: \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\") pod \"21203dac-8909-4a4d-ab9e-2fec1589db0b\" (UID: \"21203dac-8909-4a4d-ab9e-2fec1589db0b\") "
Oct 31 14:49:06.245424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.245402    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle" pod "21203dac-8909-4a4d-ab9e-2fec1589db0b" (UID: "21203dac-8909-4a4d-ab9e-2fec1589db0b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.247216 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.247195    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs" pod "21203dac-8909-4a4d-ab9e-2fec1589db0b" (UID: "21203dac-8909-4a4d-ab9e-2fec1589db0b"). InnerVolumeSpecName "kube-api-access-hmfvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:06.250450 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.250427    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util" pod "21203dac-8909-4a4d-ab9e-2fec1589db0b" (UID: "21203dac-8909-4a4d-ab9e-2fec1589db0b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.341998 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.341946    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" podStartSLOduration=2.283187648 podStartE2EDuration="48.341929795s" totalImagesPullingTime="46.058742147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:48:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:06.138223357 +0000 UTC m=+186.864504336" watchObservedRunningTime="2026-10-31 14:49:06.341929795 +0000 UTC m=+187.068210774"
Oct 31 14:49:06.345757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.345720    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\" (UniqueName: \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\") pod \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") "
Oct 31 14:49:06.345909 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.345799    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\") pod \"7c49fa35-99d4-4627-9e36-323184c01f59\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") "
Oct 31 14:49:06.345909 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.345870    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\") pod \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") "
Oct 31 14:49:06.346015 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.345935    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\") pod \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\" (UID: \"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633\") "
Oct 31 14:49:06.346015 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.345976    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\" (UniqueName: \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\") pod \"7c49fa35-99d4-4627-9e36-323184c01f59\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") "
Oct 31 14:49:06.346015 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346010    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\") pod \"7c49fa35-99d4-4627-9e36-323184c01f59\" (UID: \"7c49fa35-99d4-4627-9e36-323184c01f59\") "
Oct 31 14:49:06.346239 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346223    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.346300 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346244    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21203dac-8909-4a4d-ab9e-2fec1589db0b-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.346300 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346258    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfvs\" (UniqueName: \"kubernetes.io/projected/21203dac-8909-4a4d-ab9e-2fec1589db0b-kube-api-access-hmfvs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.346517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346492    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle" pod "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" (UID: "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.346922 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.346891    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle" pod "7c49fa35-99d4-4627-9e36-323184c01f59" (UID: "7c49fa35-99d4-4627-9e36-323184c01f59"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.348315 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.348288    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk" pod "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" (UID: "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633"). InnerVolumeSpecName "kube-api-access-2z7nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:06.348315 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.348301    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq" pod "7c49fa35-99d4-4627-9e36-323184c01f59" (UID: "7c49fa35-99d4-4627-9e36-323184c01f59"). InnerVolumeSpecName "kube-api-access-z6stq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:06.353935 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.353909    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util" pod "7c49fa35-99d4-4627-9e36-323184c01f59" (UID: "7c49fa35-99d4-4627-9e36-323184c01f59"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.354130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.354113    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util" pod "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" (UID: "d46e3d6a-66d0-44ba-8628-cd7dbd7ea633"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.446983    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.447011    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.447022    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6stq\" (UniqueName: \"kubernetes.io/projected/7c49fa35-99d4-4627-9e36-323184c01f59-kube-api-access-z6stq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.447030    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.447039    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7nk\" (UniqueName: \"kubernetes.io/projected/d46e3d6a-66d0-44ba-8628-cd7dbd7ea633-kube-api-access-2z7nk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:06.447065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:06.447047    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c49fa35-99d4-4627-9e36-323184c01f59-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:07.011187 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:49:07.011098    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:49:07.030876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.030851    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2"
Oct 31 14:49:07.030876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.030853    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71n5lh2" event={"ID":"21203dac-8909-4a4d-ab9e-2fec1589db0b","Type":"ContainerDied","Data":"60baa46e26dd9c254f14461672370a026c4a717cbe6dc8ddc4b9dc9971196687"}
Oct 31 14:49:07.031363 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.030906    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60baa46e26dd9c254f14461672370a026c4a717cbe6dc8ddc4b9dc9971196687"
Oct 31 14:49:07.032672 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.032648    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb"
Oct 31 14:49:07.032797 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.032666    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwqbzb" event={"ID":"7c49fa35-99d4-4627-9e36-323184c01f59","Type":"ContainerDied","Data":"a48ab784123849f42322470bfa70f5c3e8dd3687c2af678674ca4350c8b8f515"}
Oct 31 14:49:07.032797 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.032720    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48ab784123849f42322470bfa70f5c3e8dd3687c2af678674ca4350c8b8f515"
Oct 31 14:49:07.035003 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.034977    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerID="01eb11ee82660d72c657bb975f97fa96ae4e94746907f1eafc4d2284e788f576" exitCode=0
Oct 31 14:49:07.035131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.035052    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr" event={"ID":"bd2e0435-521b-4c35-bd44-45e57da5936e","Type":"ContainerDied","Data":"01eb11ee82660d72c657bb975f97fa96ae4e94746907f1eafc4d2284e788f576"}
Oct 31 14:49:07.036946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.036916    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4" event={"ID":"d46e3d6a-66d0-44ba-8628-cd7dbd7ea633","Type":"ContainerDied","Data":"d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5"}
Oct 31 14:49:07.036946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.036934    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f89s4"
Oct 31 14:49:07.037099 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.036951    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d06f66de2ec6de6e502022a4dfbfc1f5402904ec2f2ae15784c3f3085a5e6cc5"
Oct 31 14:49:07.039338 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.039317    4549 generic.go:356] "Generic (PLEG): container finished" podID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerID="38c53e311089b03a53cfa4ad7194938631905d06192e89a9f808560085afb459" exitCode=0
Oct 31 14:49:07.039402 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:07.039354    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerDied","Data":"38c53e311089b03a53cfa4ad7194938631905d06192e89a9f808560085afb459"}
Oct 31 14:49:08.186470 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.186443    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:49:08.189321 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.189302    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:49:08.262781 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262748    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\") pod \"3339080d-6b2c-42bc-a535-1beed62ac4e5\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") "
Oct 31 14:49:08.262955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262804    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\") pod \"3339080d-6b2c-42bc-a535-1beed62ac4e5\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") "
Oct 31 14:49:08.262955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262839    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\" (UniqueName: \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\") pod \"bd2e0435-521b-4c35-bd44-45e57da5936e\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") "
Oct 31 14:49:08.262955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262857    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\") pod \"bd2e0435-521b-4c35-bd44-45e57da5936e\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") "
Oct 31 14:49:08.262955 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262937    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\" (UniqueName: \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\") pod \"3339080d-6b2c-42bc-a535-1beed62ac4e5\" (UID: \"3339080d-6b2c-42bc-a535-1beed62ac4e5\") "
Oct 31 14:49:08.263171 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.262971    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\") pod \"bd2e0435-521b-4c35-bd44-45e57da5936e\" (UID: \"bd2e0435-521b-4c35-bd44-45e57da5936e\") "
Oct 31 14:49:08.263301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.263275    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle" pod "3339080d-6b2c-42bc-a535-1beed62ac4e5" (UID: "3339080d-6b2c-42bc-a535-1beed62ac4e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:08.263726 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.263655    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle" pod "bd2e0435-521b-4c35-bd44-45e57da5936e" (UID: "bd2e0435-521b-4c35-bd44-45e57da5936e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:08.265166 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.265134    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht" pod "bd2e0435-521b-4c35-bd44-45e57da5936e" (UID: "bd2e0435-521b-4c35-bd44-45e57da5936e"). InnerVolumeSpecName "kube-api-access-ndnht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:08.265266 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.265204    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6" pod "3339080d-6b2c-42bc-a535-1beed62ac4e5" (UID: "3339080d-6b2c-42bc-a535-1beed62ac4e5"). InnerVolumeSpecName "kube-api-access-hzpk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:49:08.271465 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.271442    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util" pod "3339080d-6b2c-42bc-a535-1beed62ac4e5" (UID: "3339080d-6b2c-42bc-a535-1beed62ac4e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:08.271672 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.271654    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util" pod "bd2e0435-521b-4c35-bd44-45e57da5936e" (UID: "bd2e0435-521b-4c35-bd44-45e57da5936e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363898    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpk6\" (UniqueName: \"kubernetes.io/projected/3339080d-6b2c-42bc-a535-1beed62ac4e5-kube-api-access-hzpk6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363927    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363937    4549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-util\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363945    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3339080d-6b2c-42bc-a535-1beed62ac4e5-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363953    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnht\" (UniqueName: \"kubernetes.io/projected/bd2e0435-521b-4c35-bd44-45e57da5936e-kube-api-access-ndnht\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:08.363978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:08.363962    4549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd2e0435-521b-4c35-bd44-45e57da5936e-bundle\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:49:09.053248 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.053221    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl"
Oct 31 14:49:09.053440 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.053215    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f2mwl" event={"ID":"3339080d-6b2c-42bc-a535-1beed62ac4e5","Type":"ContainerDied","Data":"b67d1707c4607efb2747a419a016f56e7a74cff8e3cd01dd43640dd76ac4f62c"}
Oct 31 14:49:09.053440 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.053360    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67d1707c4607efb2747a419a016f56e7a74cff8e3cd01dd43640dd76ac4f62c"
Oct 31 14:49:09.055435 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.055412    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr" event={"ID":"bd2e0435-521b-4c35-bd44-45e57da5936e","Type":"ContainerDied","Data":"352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255"}
Oct 31 14:49:09.055550 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.055452    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="352a80013a78eff0a4c9814303a353bb433035e63d98c3e2619e947b035d4255"
Oct 31 14:49:09.055550 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.055465    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgcmzr"
Oct 31 14:49:09.873217 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873183    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"]
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873484    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="util"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873495    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="util"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873504    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873508    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873514    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873519    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873529    4549 cpu_manager.go:414] "containerMap: removing container" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873533    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873540    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873544    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873551    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873555    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873561    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873565    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873588    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873596    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="extract"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873607    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="util"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873614    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="util"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873623    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873628    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873634    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="pull"
Oct 31 14:49:09.873631 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873638    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873644    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873649    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873657    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873661    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873667    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873671    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873679    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873684    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873692    4549 cpu_manager.go:414] "containerMap: removing container" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873697    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873705    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873709    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873714    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873718    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873724    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873728    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873733    4549 cpu_manager.go:414] "containerMap: removing container" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873736    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873741    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873745    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873751    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873755    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873761    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873765    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873770    4549 cpu_manager.go:414] "containerMap: removing container" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873774    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873779    4549 cpu_manager.go:414] "containerMap: removing container" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873783    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873789    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873793    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873801    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873805    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873810    4549 cpu_manager.go:414] "containerMap: removing container" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873814    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873820    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873824    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873831    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873835    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="util"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873840    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873844    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873851    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873855    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="pull"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873860    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873864    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873909    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abe7241-88d3-4dc4-bb8b-1853b4a5fc87" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873917    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44a9d6e-bf5a-4a9d-91b4-9733ed519a82" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873923    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c49fa35-99d4-4627-9e36-323184c01f59" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873929    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="21203dac-8909-4a4d-ab9e-2fec1589db0b" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873934    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f55a10a-1b2f-44f2-ab8f-8b93c94cdcc6" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873940    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f6a84-76bb-452f-9e98-6fb4225bb39d" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873944    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46e3d6a-66d0-44ba-8628-cd7dbd7ea633" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873950    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2e0435-521b-4c35-bd44-45e57da5936e" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873956    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc1e8f3-40ed-4031-9ef2-139114477860" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873961    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="af660270-8eae-4c52-9d86-de3c0e0dfde3" containerName="extract"
Oct 31 14:49:09.874372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.873966    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3339080d-6b2c-42bc-a535-1beed62ac4e5" containerName="extract"
Oct 31 14:49:09.883863 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.883845    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.940179 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.940154    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 14:49:09.940386 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.940372    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 14:49:09.940634 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.940619    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 14:49:09.940809 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.940796    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 14:49:09.940979 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.940968    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 14:49:09.941148 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.941136    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\""
Oct 31 14:49:09.978539 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978511    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.978645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978551    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.978698 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978654    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6gj\" (UniqueName: \"kubernetes.io/projected/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-kube-api-access-qx6gj\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.978744 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978701    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.978744 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978727    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:09.978815 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:09.978802    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079386 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079355    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079595 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079395    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079595 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6gj\" (UniqueName: \"kubernetes.io/projected/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-kube-api-access-qx6gj\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079598    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079628    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.079733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.079650    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.080117 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.080093    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.081884 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.081862    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.082490 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.082469    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.082733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.082710    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.082786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.082739    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.119441 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.119405    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"]
Oct 31 14:49:10.137992 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.137929    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6gj\" (UniqueName: \"kubernetes.io/projected/d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f-kube-api-access-qx6gj\") pod \"opendatahub-operator-controller-manager-c95c77854-pxgv5\" (UID: \"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.219741 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.219703    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:10.772609 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:10.772561    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f0dfb7_cbbe_4c8b_82f8_67ab5053e41f.slice/crio-8fdf277a8631feab2379ee9595b6d837911fc74550a56202d9b343945e2ea1c6 WatchSource:0}: Error finding container 8fdf277a8631feab2379ee9595b6d837911fc74550a56202d9b343945e2ea1c6: Status 404 returned error can't find the container with id 8fdf277a8631feab2379ee9595b6d837911fc74550a56202d9b343945e2ea1c6
Oct 31 14:49:10.819733 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:10.819693    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"]
Oct 31 14:49:11.067327 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:11.067243    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5" event={"ID":"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f","Type":"ContainerStarted","Data":"8fdf277a8631feab2379ee9595b6d837911fc74550a56202d9b343945e2ea1c6"}
Oct 31 14:49:15.223628 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.223594    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"]
Oct 31 14:49:15.226116 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.226101    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.291703 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.291678    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:49:15.322357 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.322309    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcmmf\" (UniqueName: \"kubernetes.io/projected/a95e44d5-896e-438a-aa30-0c4cb36db151-kube-api-access-gcmmf\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.322357 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.322361    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95e44d5-896e-438a-aa30-0c4cb36db151-tmp\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.332201 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.332168    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4b9fp\""
Oct 31 14:49:15.332301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.332212    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 14:49:15.333828 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.333812    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"]
Oct 31 14:49:15.423107 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.423032    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcmmf\" (UniqueName: \"kubernetes.io/projected/a95e44d5-896e-438a-aa30-0c4cb36db151-kube-api-access-gcmmf\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.423107 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.423067    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95e44d5-896e-438a-aa30-0c4cb36db151-tmp\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.423370 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.423353    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a95e44d5-896e-438a-aa30-0c4cb36db151-tmp\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.479989 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.479957    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcmmf\" (UniqueName: \"kubernetes.io/projected/a95e44d5-896e-438a-aa30-0c4cb36db151-kube-api-access-gcmmf\") pod \"jobset-operator-674cd59dc5-g9jkh\" (UID: \"a95e44d5-896e-438a-aa30-0c4cb36db151\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.544982 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.544953    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"
Oct 31 14:49:15.913485 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:15.913459    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh"]
Oct 31 14:49:15.915959 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:15.915935    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda95e44d5_896e_438a_aa30_0c4cb36db151.slice/crio-e4933647c61fcfd83d00b641e95651daddf97693caf2ae7fad6702f7aaa05a6f WatchSource:0}: Error finding container e4933647c61fcfd83d00b641e95651daddf97693caf2ae7fad6702f7aaa05a6f: Status 404 returned error can't find the container with id e4933647c61fcfd83d00b641e95651daddf97693caf2ae7fad6702f7aaa05a6f
Oct 31 14:49:16.096264 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:16.096226    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh" event={"ID":"a95e44d5-896e-438a-aa30-0c4cb36db151","Type":"ContainerStarted","Data":"e4933647c61fcfd83d00b641e95651daddf97693caf2ae7fad6702f7aaa05a6f"}
Oct 31 14:49:16.098344 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:16.098310    4549 generic.go:356] "Generic (PLEG): container finished" podID="d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f" containerID="e19545b7be47f976a7e32a7221a781b6eb997947cf6cb4aa58ece5003e328b2b" exitCode=0
Oct 31 14:49:16.098516 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:16.098389    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5" event={"ID":"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f","Type":"ContainerDied","Data":"e19545b7be47f976a7e32a7221a781b6eb997947cf6cb4aa58ece5003e328b2b"}
Oct 31 14:49:17.105032 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:17.104993    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5" event={"ID":"d2f0dfb7-cbbe-4c8b-82f8-67ab5053e41f","Type":"ContainerStarted","Data":"21c2a2e7ff5cfde3214feba733d27bdd8db1a46d0d9712a0bc5872bc7e248c6b"}
Oct 31 14:49:17.105504 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:17.105105    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:17.339332 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:17.339277    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5" podStartSLOduration=2.701930523 podStartE2EDuration="8.339262427s" totalImagesPullingTime="5.127612582s" totalInitContainerRuntime="509.719322ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:17.288701187 +0000 UTC m=+198.014982181" watchObservedRunningTime="2026-10-31 14:49:17.339262427 +0000 UTC m=+198.065543405"
Oct 31 14:49:19.117519 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:19.117482    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh" event={"ID":"a95e44d5-896e-438a-aa30-0c4cb36db151","Type":"ContainerStarted","Data":"698e73dc25a7e1de2eca5106861dfd26eafd3fcc68679c5e30946c7aed19cbac"}
Oct 31 14:49:19.232790 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:19.232741    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-g9jkh" podStartSLOduration=1.533618241 podStartE2EDuration="4.23272704s" totalImagesPullingTime="2.699108799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:19.230375582 +0000 UTC m=+199.956656561" watchObservedRunningTime="2026-10-31 14:49:19.23272704 +0000 UTC m=+199.959008018"
Oct 31 14:49:21.177900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.177868    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"]
Oct 31 14:49:21.184208 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.184186    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.237824 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.237796    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c265q\""
Oct 31 14:49:21.237824 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.237809    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 14:49:21.238033 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.237855    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 14:49:21.275694 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.275657    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.275850 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.275699    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm6r7\" (UniqueName: \"kubernetes.io/projected/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-kube-api-access-dm6r7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.289410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.289384    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"]
Oct 31 14:49:21.377074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.377041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.377074 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.377076    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm6r7\" (UniqueName: \"kubernetes.io/projected/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-kube-api-access-dm6r7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.377436 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.377417    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.434422 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.434346    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm6r7\" (UniqueName: \"kubernetes.io/projected/cf02357b-4771-41e8-ab2c-2af4d8f48cf9-kube-api-access-dm6r7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qv68j\" (UID: \"cf02357b-4771-41e8-ab2c-2af4d8f48cf9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.495465 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.495428    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"
Oct 31 14:49:21.974834 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:21.974805    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j"]
Oct 31 14:49:22.027490 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:22.027464    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf02357b_4771_41e8_ab2c_2af4d8f48cf9.slice/crio-45eb68bf6c269781e0ecd44226b9c590968fceea025ded0da2a406abcab3f379 WatchSource:0}: Error finding container 45eb68bf6c269781e0ecd44226b9c590968fceea025ded0da2a406abcab3f379: Status 404 returned error can't find the container with id 45eb68bf6c269781e0ecd44226b9c590968fceea025ded0da2a406abcab3f379
Oct 31 14:49:22.140897 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:22.140851    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j" event={"ID":"cf02357b-4771-41e8-ab2c-2af4d8f48cf9","Type":"ContainerStarted","Data":"45eb68bf6c269781e0ecd44226b9c590968fceea025ded0da2a406abcab3f379"}
Oct 31 14:49:25.157472 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:25.157378    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j" event={"ID":"cf02357b-4771-41e8-ab2c-2af4d8f48cf9","Type":"ContainerStarted","Data":"d090a4ed0315f59311377eef5288a9fba74cb998ca7613df8dfaff8ac497f62a"}
Oct 31 14:49:25.374496 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:25.374443    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qv68j" podStartSLOduration=1.502726096 podStartE2EDuration="4.374428204s" totalImagesPullingTime="2.871702108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:25.323450745 +0000 UTC m=+206.049731722" watchObservedRunningTime="2026-10-31 14:49:25.374428204 +0000 UTC m=+206.100709182"
Oct 31 14:49:27.402453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.402420    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"]
Oct 31 14:49:27.405992 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.405970    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.428411 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.428382    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.428544 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.428460    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hltvn\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-kube-api-access-hltvn\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.463529 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.463500    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"]
Oct 31 14:49:27.509807 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.509772    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 14:49:27.509807 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.509809    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 14:49:27.529117 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.529077    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.529378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.529193    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hltvn\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-kube-api-access-hltvn\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.590688 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.590656    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.591013 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.590993    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hltvn\" (UniqueName: \"kubernetes.io/projected/057435af-b787-4b85-83e7-79af663f78ee-kube-api-access-hltvn\") pod \"cert-manager-webhook-5dbbbf75cc-6xfbz\" (UID: \"057435af-b787-4b85-83e7-79af663f78ee\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:27.716934 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:27.716857    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:28.089845 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:28.089820    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"]
Oct 31 14:49:28.092206 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:28.092172    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod057435af_b787_4b85_83e7_79af663f78ee.slice/crio-29fe1b6cc6b1320c63412fffa4eb8789fce6152b0509a961732fc9d5503c1eb8 WatchSource:0}: Error finding container 29fe1b6cc6b1320c63412fffa4eb8789fce6152b0509a961732fc9d5503c1eb8: Status 404 returned error can't find the container with id 29fe1b6cc6b1320c63412fffa4eb8789fce6152b0509a961732fc9d5503c1eb8
Oct 31 14:49:28.112864 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:28.112839    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-pxgv5"
Oct 31 14:49:28.173172 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:28.173135    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz" event={"ID":"057435af-b787-4b85-83e7-79af663f78ee","Type":"ContainerStarted","Data":"29fe1b6cc6b1320c63412fffa4eb8789fce6152b0509a961732fc9d5503c1eb8"}
Oct 31 14:49:31.680517 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.680482    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"]
Oct 31 14:49:31.689676 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.689646    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:31.744590 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.744540    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"]
Oct 31 14:49:31.769081 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.769036    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtg2p\" (UniqueName: \"kubernetes.io/projected/fe94f527-0b08-478a-baa0-4f92659bf5d6-kube-api-access-gtg2p\") pod \"authorino-operator-7b7bb4bc77-z7xph\" (UID: \"fe94f527-0b08-478a-baa0-4f92659bf5d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:31.791997 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.791969    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 14:49:31.791997 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.791969    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 14:49:31.792216 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.792072    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-px9bk\""
Oct 31 14:49:31.870687 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.870653    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtg2p\" (UniqueName: \"kubernetes.io/projected/fe94f527-0b08-478a-baa0-4f92659bf5d6-kube-api-access-gtg2p\") pod \"authorino-operator-7b7bb4bc77-z7xph\" (UID: \"fe94f527-0b08-478a-baa0-4f92659bf5d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:31.934305 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:31.934255    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtg2p\" (UniqueName: \"kubernetes.io/projected/fe94f527-0b08-478a-baa0-4f92659bf5d6-kube-api-access-gtg2p\") pod \"authorino-operator-7b7bb4bc77-z7xph\" (UID: \"fe94f527-0b08-478a-baa0-4f92659bf5d6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:32.005669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:32.005637    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:32.195515 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:32.195428    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz" event={"ID":"057435af-b787-4b85-83e7-79af663f78ee","Type":"ContainerStarted","Data":"a655c9fed5843e56b83c76230ab3debce60442bb6daae5325c8542c6a509814c"}
Oct 31 14:49:32.195710 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:32.195550    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:32.313458 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:32.313406    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz" podStartSLOduration=1.536860607 podStartE2EDuration="5.313389237s" totalImagesPullingTime="3.77652863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:32.313338923 +0000 UTC m=+213.039619904" watchObservedRunningTime="2026-10-31 14:49:32.313389237 +0000 UTC m=+213.039670209"
Oct 31 14:49:32.472805 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:32.472778    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"]
Oct 31 14:49:32.474403 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:32.474370    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe94f527_0b08_478a_baa0_4f92659bf5d6.slice/crio-9d860df54bde90f4d496fda64e34f24f3fc20dcb848ee2b863808b6b17ed458c WatchSource:0}: Error finding container 9d860df54bde90f4d496fda64e34f24f3fc20dcb848ee2b863808b6b17ed458c: Status 404 returned error can't find the container with id 9d860df54bde90f4d496fda64e34f24f3fc20dcb848ee2b863808b6b17ed458c
Oct 31 14:49:33.204235 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:33.204206    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph" event={"ID":"fe94f527-0b08-478a-baa0-4f92659bf5d6","Type":"ContainerStarted","Data":"9d860df54bde90f4d496fda64e34f24f3fc20dcb848ee2b863808b6b17ed458c"}
Oct 31 14:49:36.222444 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:36.222357    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph" event={"ID":"fe94f527-0b08-478a-baa0-4f92659bf5d6","Type":"ContainerStarted","Data":"a1b88037a6c541c1769b8d55b8fdcc021bfe7fb0860d849527ea2f39fe5a8600"}
Oct 31 14:49:36.222843 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:36.222485    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:36.345516 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:36.345463    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph" podStartSLOduration=2.024237465 podStartE2EDuration="5.34545001s" totalImagesPullingTime="3.321212545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:36.344858699 +0000 UTC m=+217.071139689" watchObservedRunningTime="2026-10-31 14:49:36.34545001 +0000 UTC m=+217.071730987"
Oct 31 14:49:37.020177 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:49:37.020127    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:49:38.206353 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:38.206326    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz"
Oct 31 14:49:39.079300 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.079263    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 14:49:39.083829 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.083809    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.129684 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.129652    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtvt\" (UniqueName: \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.129845 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.129708    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.147648 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.147626    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjrh5\""
Oct 31 14:49:39.188491 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.188458    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 14:49:39.230980 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.230941    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.231356 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.231029    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtvt\" (UniqueName: \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.231396 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.231354    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.287582 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.287541    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtvt\" (UniqueName: \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\") pod \"kuadrant-operator-controller-manager-678d87766f-b7wgz\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.396140 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.396057    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:39.873615 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:39.873556    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 14:49:39.875755 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:39.875719    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb301180c_09ce_4d9b_abf3_45e80856e298.slice/crio-193de42fdccd7366a6341d70602d64f92ccea690059f0055bb7d46ac54eea56b WatchSource:0}: Error finding container 193de42fdccd7366a6341d70602d64f92ccea690059f0055bb7d46ac54eea56b: Status 404 returned error can't find the container with id 193de42fdccd7366a6341d70602d64f92ccea690059f0055bb7d46ac54eea56b
Oct 31 14:49:40.242949 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:40.242871    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" event={"ID":"b301180c-09ce-4d9b-abf3-45e80856e298","Type":"ContainerStarted","Data":"193de42fdccd7366a6341d70602d64f92ccea690059f0055bb7d46ac54eea56b"}
Oct 31 14:49:42.103671 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.103617    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"]
Oct 31 14:49:42.107054 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.107033    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:42.158687 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.158649    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4sr\" (UniqueName: \"kubernetes.io/projected/fa6cac91-ffce-4ec7-8c7d-1fc35213fef7-kube-api-access-gw4sr\") pod \"dns-operator-controller-manager-8557cbdb84-52jvs\" (UID: \"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:42.215138 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.215095    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 14:49:42.215138 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.215126    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g2n6k\""
Oct 31 14:49:42.217957 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.217933    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"]
Oct 31 14:49:42.259677 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.259641    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4sr\" (UniqueName: \"kubernetes.io/projected/fa6cac91-ffce-4ec7-8c7d-1fc35213fef7-kube-api-access-gw4sr\") pod \"dns-operator-controller-manager-8557cbdb84-52jvs\" (UID: \"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:42.317480 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.317442    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4sr\" (UniqueName: \"kubernetes.io/projected/fa6cac91-ffce-4ec7-8c7d-1fc35213fef7-kube-api-access-gw4sr\") pod \"dns-operator-controller-manager-8557cbdb84-52jvs\" (UID: \"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:42.422943 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.422854    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:42.807340 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:42.807306    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"]
Oct 31 14:49:42.841351 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:42.841311    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6cac91_ffce_4ec7_8c7d_1fc35213fef7.slice/crio-49bd1a6603909340f6f40ef483af8f45367eb70679a768192c4fded1a55b934d WatchSource:0}: Error finding container 49bd1a6603909340f6f40ef483af8f45367eb70679a768192c4fded1a55b934d: Status 404 returned error can't find the container with id 49bd1a6603909340f6f40ef483af8f45367eb70679a768192c4fded1a55b934d
Oct 31 14:49:43.261688 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:43.261600    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" event={"ID":"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7","Type":"ContainerStarted","Data":"49bd1a6603909340f6f40ef483af8f45367eb70679a768192c4fded1a55b934d"}
Oct 31 14:49:45.274809 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:45.274769    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" event={"ID":"b301180c-09ce-4d9b-abf3-45e80856e298","Type":"ContainerStarted","Data":"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"}
Oct 31 14:49:45.275232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:45.274837    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:45.499098 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:45.499045    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" podStartSLOduration=2.7297404739999998 podStartE2EDuration="7.49902698s" totalImagesPullingTime="4.769286506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:45.447226257 +0000 UTC m=+226.173507237" watchObservedRunningTime="2026-10-31 14:49:45.49902698 +0000 UTC m=+226.225307961"
Oct 31 14:49:46.282176 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.282131    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" event={"ID":"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7","Type":"ContainerStarted","Data":"5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"}
Oct 31 14:49:46.282699 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.282295    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:49:46.406857 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.406802    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podStartSLOduration=1.8188942670000001 podStartE2EDuration="4.406788225s" totalImagesPullingTime="2.587893958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:46.403753193 +0000 UTC m=+227.130034171" watchObservedRunningTime="2026-10-31 14:49:46.406788225 +0000 UTC m=+227.133069204"
Oct 31 14:49:46.840773 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.840742    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"]
Oct 31 14:49:46.844489 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.844472    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.896174 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.896147    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sn8hj\""
Oct 31 14:49:46.903335 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903300    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903490 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903349    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlfm\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-kube-api-access-ttlfm\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903490 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903445    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"]
Oct 31 14:49:46.903490 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903457    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903719 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903495    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903764    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08359732-c9bd-4176-94b3-f869c31849a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903853 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903833    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08359732-c9bd-4176-94b3-f869c31849a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.903978 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.903956    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.904044 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.904028    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:46.904093 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:46.904057    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004554 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004522    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004598    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004616    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004641    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004660    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlfm\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-kube-api-access-ttlfm\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004685    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004703    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.004835 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004738    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08359732-c9bd-4176-94b3-f869c31849a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004842    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08359732-c9bd-4176-94b3-f869c31849a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.004985    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.005080    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.005144    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.005227    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.005776 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.005592    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08359732-c9bd-4176-94b3-f869c31849a4-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.008273 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.008248    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08359732-c9bd-4176-94b3-f869c31849a4-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.008751 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.008731    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08359732-c9bd-4176-94b3-f869c31849a4-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.065996 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.065964    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.068266 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.068245    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlfm\" (UniqueName: \"kubernetes.io/projected/08359732-c9bd-4176-94b3-f869c31849a4-kube-api-access-ttlfm\") pod \"data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b\" (UID: \"08359732-c9bd-4176-94b3-f869c31849a4\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.157193 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.157112    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:47.229892 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.229863    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-z7xph"
Oct 31 14:49:47.582859 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.582832    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"]
Oct 31 14:49:47.584947 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:49:47.584919    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08359732_c9bd_4176_94b3_f869c31849a4.slice/crio-be6b971f46e635384d72be2a4abf744bb63be7ea23763f17214086ab3e1c8eb4 WatchSource:0}: Error finding container be6b971f46e635384d72be2a4abf744bb63be7ea23763f17214086ab3e1c8eb4: Status 404 returned error can't find the container with id be6b971f46e635384d72be2a4abf744bb63be7ea23763f17214086ab3e1c8eb4
Oct 31 14:49:47.587132 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.587090    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:49:47.587243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.587168    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:49:47.587243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:47.587200    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:49:48.293382 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:48.293350    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b" event={"ID":"08359732-c9bd-4176-94b3-f869c31849a4","Type":"ContainerStarted","Data":"26687c78ac55d5a2cf6475a2ecc05ad2efc8bd3d8ca8712d60b4fd6e1780fc44"}
Oct 31 14:49:48.293557 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:48.293403    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b" event={"ID":"08359732-c9bd-4176-94b3-f869c31849a4","Type":"ContainerStarted","Data":"be6b971f46e635384d72be2a4abf744bb63be7ea23763f17214086ab3e1c8eb4"}
Oct 31 14:49:48.409564 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:48.409514    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b" podStartSLOduration=2.409497746 podStartE2EDuration="2.409497746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:48.40804755 +0000 UTC m=+229.134328540" watchObservedRunningTime="2026-10-31 14:49:48.409497746 +0000 UTC m=+229.135778723"
Oct 31 14:49:49.157467 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:49.157429    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:49.161901 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:49.161874    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:49.298775 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:49.298745    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:49.299670 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:49.299646    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b"
Oct 31 14:49:56.284938 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:56.284905    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 14:49:57.289443 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:49:57.289409    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 14:50:00.312185 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.312146    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"]
Oct 31 14:50:00.321658 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.321633    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.372351 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.372320    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"]
Oct 31 14:50:00.419685 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.419657    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 14:50:00.422842 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.422821    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4jx6s\""
Oct 31 14:50:00.422972 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.422856    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 14:50:00.422972 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.422891    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 14:50:00.435164 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.435130    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-cert\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.435377 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.435174    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gh5\" (UniqueName: \"kubernetes.io/projected/206ba35a-15a9-46f1-b437-582487cb1b8d-kube-api-access-54gh5\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.435377 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.435249    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/206ba35a-15a9-46f1-b437-582487cb1b8d-manager-config\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.435533 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.435438    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-metrics-certs\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.536120 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.536075    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-metrics-certs\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.536330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.536130    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-cert\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.536330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.536163    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gh5\" (UniqueName: \"kubernetes.io/projected/206ba35a-15a9-46f1-b437-582487cb1b8d-kube-api-access-54gh5\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.536330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.536220    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/206ba35a-15a9-46f1-b437-582487cb1b8d-manager-config\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.536959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.536937    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/206ba35a-15a9-46f1-b437-582487cb1b8d-manager-config\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.538818 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.538801    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-metrics-certs\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.538882 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.538849    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/206ba35a-15a9-46f1-b437-582487cb1b8d-cert\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.593432 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.593350    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gh5\" (UniqueName: \"kubernetes.io/projected/206ba35a-15a9-46f1-b437-582487cb1b8d-kube-api-access-54gh5\") pod \"jobset-controller-manager-5f948884b-n2v4c\" (UID: \"206ba35a-15a9-46f1-b437-582487cb1b8d\") " pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:00.634010 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:00.633976    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:01.012987 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:01.010014    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"]
Oct 31 14:50:01.360540 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:01.360503    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c" event={"ID":"206ba35a-15a9-46f1-b437-582487cb1b8d","Type":"ContainerStarted","Data":"f7a8ce9c5869f6104f4c701c3b1de6b2725c5b8b7478b44f9a8ca338c083d86f"}
Oct 31 14:50:07.012946 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:50:07.012909    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:50:07.397798 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:07.397757    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c" event={"ID":"206ba35a-15a9-46f1-b437-582487cb1b8d","Type":"ContainerStarted","Data":"68d49a625a255be09b6198376e41174b026dd6d63941dcce1bf1ca87dcabf1f7"}
Oct 31 14:50:07.397982 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:07.397853    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:07.510116 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:07.510061    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c" podStartSLOduration=2.048805653 podStartE2EDuration="7.510043114s" totalImagesPullingTime="5.461237461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:07.509683219 +0000 UTC m=+248.235964198" watchObservedRunningTime="2026-10-31 14:50:07.510043114 +0000 UTC m=+248.236324092"
Oct 31 14:50:18.408376 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:18.408350    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 14:50:37.011247 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:50:37.011206    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:50:48.898155 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.898123    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:50:48.902700 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.902676    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:48.939139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.939103    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:48.939139 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.939139    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cjj\" (UniqueName: \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:48.939339 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.939274    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:48.954934 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:48.954911    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 14:50:49.004387 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.004361    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 14:50:49.004633 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.004607    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 14:50:49.004804 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.004654    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 14:50:49.004804 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.004667    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 14:50:49.004920 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.004682    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-b5bsw\""
Oct 31 14:50:49.006950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.006933    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:50:49.040442 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.040407    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.040664 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.040471    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cjj\" (UniqueName: \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.040664 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.040583    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.043358 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.043332    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.043508 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.043329    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.097848 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.097811    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cjj\" (UniqueName: \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\") pod \"maas-controller-658c75645b-6f6zf\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") " pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.216829 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.216738    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:49.468937 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.468862    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 14:50:49.474342 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.474322    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.532503 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.532473    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 14:50:49.546111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.546081    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.546280 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.546120    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.546280 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.546143    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq475\" (UniqueName: \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.647132 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.647101    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.647313 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.647142    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.647313 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.647168    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq475\" (UniqueName: \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.650200 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.650172    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.650871 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.650849    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.652649 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.652629    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:50:49.654363 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:50:49.654340    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0f47f2_9b54_4620_9b4d_f95d7369ca6e.slice/crio-5fe34e910ab530f8674bd02d425fc6eceaa9aee563c54e135c79014d02fa2acd WatchSource:0}: Error finding container 5fe34e910ab530f8674bd02d425fc6eceaa9aee563c54e135c79014d02fa2acd: Status 404 returned error can't find the container with id 5fe34e910ab530f8674bd02d425fc6eceaa9aee563c54e135c79014d02fa2acd
Oct 31 14:50:49.704253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.704221    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq475\" (UniqueName: \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\") pod \"maas-controller-dff98979c-sr98m\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") " pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:49.788225 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:49.788199    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:50.159229 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:50.159202    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 14:50:50.160705 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:50:50.160678    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f43ebaf_2e1d_4f28_9e85_71f708747990.slice/crio-2d48dd817e36321fd953dfe8314ba9bcdd1603e59dd689cf5d9391c4a864ead0 WatchSource:0}: Error finding container 2d48dd817e36321fd953dfe8314ba9bcdd1603e59dd689cf5d9391c4a864ead0: Status 404 returned error can't find the container with id 2d48dd817e36321fd953dfe8314ba9bcdd1603e59dd689cf5d9391c4a864ead0
Oct 31 14:50:50.613248 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:50.613206    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-sr98m" event={"ID":"4f43ebaf-2e1d-4f28-9e85-71f708747990","Type":"ContainerStarted","Data":"2d48dd817e36321fd953dfe8314ba9bcdd1603e59dd689cf5d9391c4a864ead0"}
Oct 31 14:50:50.614258 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:50.614237    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-658c75645b-6f6zf" event={"ID":"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e","Type":"ContainerStarted","Data":"5fe34e910ab530f8674bd02d425fc6eceaa9aee563c54e135c79014d02fa2acd"}
Oct 31 14:50:56.652123 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.652087    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-sr98m" event={"ID":"4f43ebaf-2e1d-4f28-9e85-71f708747990","Type":"ContainerStarted","Data":"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"}
Oct 31 14:50:56.652595 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.652157    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:50:56.653735 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.653704    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-658c75645b-6f6zf" event={"ID":"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e","Type":"ContainerStarted","Data":"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"}
Oct 31 14:50:56.653828 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.653785    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:50:56.766106 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.766034    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-dff98979c-sr98m" podStartSLOduration=2.316059242 podStartE2EDuration="7.766004487s" totalImagesPullingTime="5.449945245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:56.765614366 +0000 UTC m=+297.491895349" watchObservedRunningTime="2026-10-31 14:50:56.766004487 +0000 UTC m=+297.492285465"
Oct 31 14:50:56.881278 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:56.881225    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-658c75645b-6f6zf" podStartSLOduration=2.914882741 podStartE2EDuration="8.881209949s" totalImagesPullingTime="5.966327208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:56.880243353 +0000 UTC m=+297.606524330" watchObservedRunningTime="2026-10-31 14:50:56.881209949 +0000 UTC m=+297.607490975"
Oct 31 14:50:59.430797 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:59.430770    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:50:59.435067 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:50:59.435037    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:51:07.014017 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:51:07.013972    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:51:07.661959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:07.661927    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 14:51:07.662122 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:07.662013    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:51:07.850401 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:07.850362    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:51:07.850658 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:07.850610    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-658c75645b-6f6zf" podUID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" containerName="manager" containerID="cri-o://8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64" gracePeriod=30
Oct 31 14:51:08.112378 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.112355    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:51:08.221873 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.221794    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\" (UniqueName: \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\") pod \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") "
Oct 31 14:51:08.221873 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.221829    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\") pod \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") "
Oct 31 14:51:08.221873 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.221874    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\") pod \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\" (UID: \"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e\") "
Oct 31 14:51:08.224154 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.224126    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj" pod "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" (UID: "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e"). InnerVolumeSpecName "kube-api-access-78cjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:51:08.224275 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.224162    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs" pod "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" (UID: "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:51:08.224275 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.224205    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert" pod "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" (UID: "5d0f47f2-9b54-4620-9b4d-f95d7369ca6e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:51:08.323036 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.322998    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cjj\" (UniqueName: \"kubernetes.io/projected/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-kube-api-access-78cjj\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:51:08.323036 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.323030    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:51:08.323036 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.323040    4549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e-webhook-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:51:08.723349 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.723311    4549 generic.go:356] "Generic (PLEG): container finished" podID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" containerID="8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64" exitCode=0
Oct 31 14:51:08.723525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.723366    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-658c75645b-6f6zf"
Oct 31 14:51:08.723525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.723377    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-658c75645b-6f6zf" event={"ID":"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e","Type":"ContainerDied","Data":"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"}
Oct 31 14:51:08.723525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.723441    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-658c75645b-6f6zf" event={"ID":"5d0f47f2-9b54-4620-9b4d-f95d7369ca6e","Type":"ContainerDied","Data":"5fe34e910ab530f8674bd02d425fc6eceaa9aee563c54e135c79014d02fa2acd"}
Oct 31 14:51:08.723525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.723461    4549 scope.go:122] "RemoveContainer" containerID="8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"
Oct 31 14:51:08.733084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.733064    4549 scope.go:122] "RemoveContainer" containerID="8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"
Oct 31 14:51:08.733369 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:51:08.733347    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64\": container with ID starting with 8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64 not found: ID does not exist" containerID="8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"
Oct 31 14:51:08.733412 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.733380    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64"} err="failed to get container status \"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64\": rpc error: code = NotFound desc = could not find container \"8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64\": container with ID starting with 8739cfa1f928aa8ff1c2bdd36f3188894ac056acadd251d5aad26336d7722a64 not found: ID does not exist"
Oct 31 14:51:08.887130 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.887100    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:51:08.936722 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:08.936691    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-658c75645b-6f6zf"]
Oct 31 14:51:09.417403 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:09.417373    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" path="/var/lib/kubelet/pods/5d0f47f2-9b54-4620-9b4d-f95d7369ca6e/volumes"
Oct 31 14:51:19.109638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.109541    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:51:19.110027 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.109970    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" containerName="manager"
Oct 31 14:51:19.110027 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.109984    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" containerName="manager"
Oct 31 14:51:19.110108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.110056    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0f47f2-9b54-4620-9b4d-f95d7369ca6e" containerName="manager"
Oct 31 14:51:19.115521 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.115501    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.169537 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.169502    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:51:19.216505 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.216470    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xhl5g\""
Oct 31 14:51:19.216505 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.216471    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 14:51:19.230946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.230916    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.231124 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.230981    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzcm\" (UniqueName: \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.332350 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.332312    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.332549 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.332394    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzcm\" (UniqueName: \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.334992 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.334959    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.392755 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.392693    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzcm\" (UniqueName: \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\") pod \"llmisvc-controller-manager-869d5dcc5d-5gpd2\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.429305 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.429271    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:19.817368 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.817327    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:51:19.821116 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:51:19.821079    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb4e4598_9abb_4a1d_8611_d4563df7f11d.slice/crio-2a2898cc770e86b36121976963a47b806bd8176ea84d8af63e7726a7fc36e4bb WatchSource:0}: Error finding container 2a2898cc770e86b36121976963a47b806bd8176ea84d8af63e7726a7fc36e4bb: Status 404 returned error can't find the container with id 2a2898cc770e86b36121976963a47b806bd8176ea84d8af63e7726a7fc36e4bb
Oct 31 14:51:19.822478 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:19.822457    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:51:20.786945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:20.786910    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" event={"ID":"eb4e4598-9abb-4a1d-8611-d4563df7f11d","Type":"ContainerStarted","Data":"2a2898cc770e86b36121976963a47b806bd8176ea84d8af63e7726a7fc36e4bb"}
Oct 31 14:51:23.807759 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:23.807718    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" event={"ID":"eb4e4598-9abb-4a1d-8611-d4563df7f11d","Type":"ContainerStarted","Data":"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"}
Oct 31 14:51:23.808223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:23.808000    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:51:23.922285 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:23.922233    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" podStartSLOduration=1.737737044 podStartE2EDuration="4.92221643s" totalImagesPullingTime="3.184479386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:23.920776653 +0000 UTC m=+324.647057622" watchObservedRunningTime="2026-10-31 14:51:23.92221643 +0000 UTC m=+324.648497408"
Oct 31 14:51:37.011314 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:51:37.011273    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:51:47.957301 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.957271    4549 generic.go:356] "Generic (PLEG): container finished" podID="3375e0e0-48b4-4232-95af-66e6751482f0" containerID="85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd" exitCode=0
Oct 31 14:51:47.957764 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.957352    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerDied","Data":"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"}
Oct 31 14:51:47.957764 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.957732    4549 scope.go:122] "RemoveContainer" containerID="85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"
Oct 31 14:51:47.958946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.958913    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:51:47.959017 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.959001    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:51:47.959066 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:47.959052    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:51:48.964821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:48.964790    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerStarted","Data":"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"}
Oct 31 14:51:49.222889 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:49.222785    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:51:49.223982 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:49.223956    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 14:51:49.224131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:49.224005    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 14:51:50.222731 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:50.222686    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 14:51:50.223214 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:50.222744    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 14:51:50.389382 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:50.389351    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:51:51.222996 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:51.222942    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 14:51:51.223468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:51.223021    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 14:51:52.222594 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:52.221985    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 14:51:52.222594 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:52.222046    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 14:51:53.222019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:53.221982    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Oct 31 14:51:53.222633 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:53.222592    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Oct 31 14:51:54.225847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:54.225816    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:51:54.226833 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:54.226816    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 14:51:54.815368 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:51:54.815339    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:52:07.011048 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:52:07.011010    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:52:33.734506 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:33.734459    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"]
Oct 31 14:52:33.738082 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:33.738065    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:33.794513 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:33.794485    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"]
Oct 31 14:52:33.925147 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:33.925107    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqp7v\" (UniqueName: \"kubernetes.io/projected/fac77ca4-d950-4750-9ed3-3c8550b6549e-kube-api-access-wqp7v\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:33.925335 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:33.925166    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac77ca4-d950-4750-9ed3-3c8550b6549e-cert\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.026456 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.026420    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqp7v\" (UniqueName: \"kubernetes.io/projected/fac77ca4-d950-4750-9ed3-3c8550b6549e-kube-api-access-wqp7v\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.026651 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.026481    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac77ca4-d950-4750-9ed3-3c8550b6549e-cert\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.029106 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.029078    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fac77ca4-d950-4750-9ed3-3c8550b6549e-cert\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.083189 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.083159    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqp7v\" (UniqueName: \"kubernetes.io/projected/fac77ca4-d950-4750-9ed3-3c8550b6549e-kube-api-access-wqp7v\") pod \"llmisvc-controller-manager-6466c554d8-fc9km\" (UID: \"fac77ca4-d950-4750-9ed3-3c8550b6549e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.350969 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.350864    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:34.723244 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:34.723216    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"]
Oct 31 14:52:34.725117 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:52:34.725091    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac77ca4_d950_4750_9ed3_3c8550b6549e.slice/crio-eb7e332d555fa4f69508d2110603208b5348687a8c0a7818ea8ce6e642fb6bc8 WatchSource:0}: Error finding container eb7e332d555fa4f69508d2110603208b5348687a8c0a7818ea8ce6e642fb6bc8: Status 404 returned error can't find the container with id eb7e332d555fa4f69508d2110603208b5348687a8c0a7818ea8ce6e642fb6bc8
Oct 31 14:52:35.202333 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:35.202300    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km" event={"ID":"fac77ca4-d950-4750-9ed3-3c8550b6549e","Type":"ContainerStarted","Data":"eb7e332d555fa4f69508d2110603208b5348687a8c0a7818ea8ce6e642fb6bc8"}
Oct 31 14:52:36.209896 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:36.209864    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km" event={"ID":"fac77ca4-d950-4750-9ed3-3c8550b6549e","Type":"ContainerStarted","Data":"25ddafb3fb8ad5bc37996cf96bc281a6971858774a8e30d5211a92c2294217f0"}
Oct 31 14:52:36.210302 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:36.209953    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:52:36.324493 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:52:36.324442    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km" podStartSLOduration=2.529370041 podStartE2EDuration="3.324428297s" totalImagesPullingTime="795.058256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:52:36.323267243 +0000 UTC m=+397.049548222" watchObservedRunningTime="2026-10-31 14:52:36.324428297 +0000 UTC m=+397.050709275"
Oct 31 14:52:37.011457 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:52:37.011419    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:53:07.011040 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:53:07.010998    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:53:07.218063 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.218033    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-fc9km"
Oct 31 14:53:07.386471 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.386437    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:53:07.386712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.386674    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" podUID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" containerName="manager" containerID="cri-o://7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1" gracePeriod=30
Oct 31 14:53:07.639135 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.639072    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:53:07.717471 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.717432    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\" (UniqueName: \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\") pod \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") "
Oct 31 14:53:07.717471 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.717471    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\" (UniqueName: \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\") pod \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\" (UID: \"eb4e4598-9abb-4a1d-8611-d4563df7f11d\") "
Oct 31 14:53:07.719783 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.719748    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert" pod "eb4e4598-9abb-4a1d-8611-d4563df7f11d" (UID: "eb4e4598-9abb-4a1d-8611-d4563df7f11d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:53:07.719904 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.719801    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm" pod "eb4e4598-9abb-4a1d-8611-d4563df7f11d" (UID: "eb4e4598-9abb-4a1d-8611-d4563df7f11d"). InnerVolumeSpecName "kube-api-access-vwzcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:53:07.818340 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.818303    4549 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb4e4598-9abb-4a1d-8611-d4563df7f11d-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:53:07.818340 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:07.818334    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzcm\" (UniqueName: \"kubernetes.io/projected/eb4e4598-9abb-4a1d-8611-d4563df7f11d-kube-api-access-vwzcm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:53:08.375501 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.375467    4549 generic.go:356] "Generic (PLEG): container finished" podID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" containerID="7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1" exitCode=0
Oct 31 14:53:08.376030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.375520    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"
Oct 31 14:53:08.376030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.375553    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" event={"ID":"eb4e4598-9abb-4a1d-8611-d4563df7f11d","Type":"ContainerDied","Data":"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"}
Oct 31 14:53:08.376030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.375635    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2" event={"ID":"eb4e4598-9abb-4a1d-8611-d4563df7f11d","Type":"ContainerDied","Data":"2a2898cc770e86b36121976963a47b806bd8176ea84d8af63e7726a7fc36e4bb"}
Oct 31 14:53:08.376030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.375658    4549 scope.go:122] "RemoveContainer" containerID="7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"
Oct 31 14:53:08.384965 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.384945    4549 scope.go:122] "RemoveContainer" containerID="7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"
Oct 31 14:53:08.385257 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:53:08.385238    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1\": container with ID starting with 7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1 not found: ID does not exist" containerID="7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"
Oct 31 14:53:08.385309 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.385266    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1"} err="failed to get container status \"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1\": rpc error: code = NotFound desc = could not find container \"7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1\": container with ID starting with 7b6abc6df720bbfd95dc76aa28bd3e5aed4e4dc3a83b28adb5a5d732455029e1 not found: ID does not exist"
Oct 31 14:53:08.541188 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.541159    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:53:08.547362 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:08.547336    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5gpd2"]
Oct 31 14:53:09.417394 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:53:09.417365    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" path="/var/lib/kubelet/pods/eb4e4598-9abb-4a1d-8611-d4563df7f11d/volumes"
Oct 31 14:53:37.011397 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:53:37.011359    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:54:07.013129 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:54:07.013086    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:54:37.011194 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:54:37.011101    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:54:37.843759 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.843714    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 14:54:37.844095 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.844083    4549 cpu_manager.go:414] "containerMap: removing container" podUID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" containerName="manager"
Oct 31 14:54:37.844147 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.844097    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" containerName="manager"
Oct 31 14:54:37.844184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.844168    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4e4598-9abb-4a1d-8611-d4563df7f11d" containerName="manager"
Oct 31 14:54:37.847095 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.847079    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916271 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916233    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49br\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916281    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916367    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916396    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916438 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916429    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916601 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916519    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916639 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916602    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916677 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916655    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:37.916716 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:37.916686    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.017738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017692    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.017738 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017741    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017797    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017826    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017885    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49br\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017924    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017950    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.017973    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.018003    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.018339    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.018366    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.018449    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.018659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.018533    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.019035 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.019009    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.020108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.020089    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.020397 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.020379    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.049501 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.049467    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 14:54:38.074026 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.073990    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.074167 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.074089    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49br\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\") pod \"maas-default-gateway-openshift-default-67bf986f99-4q2wz\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.159687 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.159604    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:38.583207 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.583180    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 14:54:38.584622 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:54:38.584590    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2005099_85d2_48d0_961c_4688729b9099.slice/crio-11ad460425e93671c76177ceeffa648dc9f3062c1fa944ae0098bc5c7097d5c8 WatchSource:0}: Error finding container 11ad460425e93671c76177ceeffa648dc9f3062c1fa944ae0098bc5c7097d5c8: Status 404 returned error can't find the container with id 11ad460425e93671c76177ceeffa648dc9f3062c1fa944ae0098bc5c7097d5c8
Oct 31 14:54:38.586682 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.586648    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:54:38.586786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.586730    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:54:38.586786 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.586776    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:54:38.878942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.878846    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" event={"ID":"a2005099-85d2-48d0-961c-4688729b9099","Type":"ContainerStarted","Data":"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"}
Oct 31 14:54:38.878942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.878897    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" event={"ID":"a2005099-85d2-48d0-961c-4688729b9099","Type":"ContainerStarted","Data":"11ad460425e93671c76177ceeffa648dc9f3062c1fa944ae0098bc5c7097d5c8"}
Oct 31 14:54:38.996597 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:38.996513    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podStartSLOduration=1.996496316 podStartE2EDuration="1.996496316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:38.993940887 +0000 UTC m=+519.720221867" watchObservedRunningTime="2026-10-31 14:54:38.996496316 +0000 UTC m=+519.722777294"
Oct 31 14:54:39.159950 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:39.159860    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:39.161383 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:39.161351    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:39.161505 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:39.161401    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:40.160332 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:40.160295    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:40.160762 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:40.160351    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:41.160544 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:41.160510    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:41.160973 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:41.160601    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:42.160822 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:42.160782    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:42.161233 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:42.160836    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:43.161002 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:43.160966    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:43.161398 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:43.161031    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:44.160242 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:44.160210    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:44.160418 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:44.160264    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:45.160397 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:45.160364    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:45.160789 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:45.160416    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:46.160388 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:46.160350    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 31 14:54:46.160555 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:46.160444    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 31 14:54:47.778980 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:47.778951    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:47.780077 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:47.780044    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:54:47.780591 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:54:47.780557    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 14:55:07.010866 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:55:07.010828    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:55:33.728695 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.728663    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:55:33.732224 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.732208    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.790019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.789977    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:55:33.836483 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.836453    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-33bc513e\""
Oct 31 14:55:33.836683 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.836627    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-33bc513e-dockercfg-7tgs2\""
Oct 31 14:55:33.837253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.837232    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.837408 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.837392    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgcf\" (UniqueName: \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.938897 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.938863    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.939160 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.939134    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgcf\" (UniqueName: \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.940149 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.940124    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:33.995986 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:33.995912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgcf\" (UniqueName: \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\") pod \"payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:34.045044 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:34.045016    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:34.422143 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:34.422117    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:55:34.423815 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:55:34.423786    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab58c94_fd19_48fe_a5f8_892c4050bdd1.slice/crio-c3926928954cdd351dd0b044d89146aeb78bd52b678cc441feb5a52a7d08043e WatchSource:0}: Error finding container c3926928954cdd351dd0b044d89146aeb78bd52b678cc441feb5a52a7d08043e: Status 404 returned error can't find the container with id c3926928954cdd351dd0b044d89146aeb78bd52b678cc441feb5a52a7d08043e
Oct 31 14:55:35.199946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:35.199912    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" event={"ID":"dab58c94-fd19-48fe-a5f8-892c4050bdd1","Type":"ContainerStarted","Data":"c3926928954cdd351dd0b044d89146aeb78bd52b678cc441feb5a52a7d08043e"}
Oct 31 14:55:36.265692 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.265652    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:55:36.274694 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.274665    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.338975 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.338940    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:55:36.362907 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.362865    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqj2c\" (UniqueName: \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.363108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.362946    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.371439 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.371404    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b671479d\""
Oct 31 14:55:36.371635 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.371465    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b671479d-dockercfg-t7bsk\""
Oct 31 14:55:36.463495 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.463456    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.463740 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.463614    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqj2c\" (UniqueName: \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.464391 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.464369    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.520653 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.520581    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqj2c\" (UniqueName: \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\") pod \"payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.591235 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.591189    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:36.974194 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:36.974168    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:55:36.975225 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:55:36.975198    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b173150_d48c_4360_8372_3ee84883e994.slice/crio-228e09d6e0558b4267b4db8925779202a4e13a9d2969cbaa3e62aaf554661a1c WatchSource:0}: Error finding container 228e09d6e0558b4267b4db8925779202a4e13a9d2969cbaa3e62aaf554661a1c: Status 404 returned error can't find the container with id 228e09d6e0558b4267b4db8925779202a4e13a9d2969cbaa3e62aaf554661a1c
Oct 31 14:55:37.020773 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:55:37.020733    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:55:37.213475 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:37.213444    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" event={"ID":"2b173150-d48c-4360-8372-3ee84883e994","Type":"ContainerStarted","Data":"228e09d6e0558b4267b4db8925779202a4e13a9d2969cbaa3e62aaf554661a1c"}
Oct 31 14:55:37.214994 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:37.214969    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" event={"ID":"dab58c94-fd19-48fe-a5f8-892c4050bdd1","Type":"ContainerStarted","Data":"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"}
Oct 31 14:55:37.215111 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:37.215100    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:37.328132 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:37.328081    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" podStartSLOduration=1.724725268 podStartE2EDuration="4.328065235s" totalImagesPullingTime="2.603339967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:37.326943831 +0000 UTC m=+578.053224811" watchObservedRunningTime="2026-10-31 14:55:37.328065235 +0000 UTC m=+578.054346213"
Oct 31 14:55:38.229279 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:38.229226    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" event={"ID":"2b173150-d48c-4360-8372-3ee84883e994","Type":"ContainerStarted","Data":"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"}
Oct 31 14:55:38.229503 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:38.229483    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:38.349857 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:38.349797    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" podStartSLOduration=1.576201934 podStartE2EDuration="2.349779744s" totalImagesPullingTime="773.57781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:38.346927135 +0000 UTC m=+579.073208134" watchObservedRunningTime="2026-10-31 14:55:38.349779744 +0000 UTC m=+579.076060725"
Oct 31 14:55:38.997253 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:38.997211    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:55:39.000878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.000861    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.090372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.090336    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.090372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.090370    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.090565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.090394    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdt9\" (UniqueName: \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.090565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.090531    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.104805 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.104774    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:55:39.104970 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.104852    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 14:55:39.104970 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.104850    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:55:39.104970 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.104906    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b1c2330c-serving-cert\""
Oct 31 14:55:39.104970 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.104939    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 14:55:39.107373 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.107355    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:55:39.191312 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.191273    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.191312 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.191310    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.191565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.191512    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdt9\" (UniqueName: \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.191661 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.191598    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.191788 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:55:39.191759    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-serving-cert: secret "maas-api-e2e-migrate-b1c2330c-serving-cert" not found
Oct 31 14:55:39.191912 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:55:39.191831    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls podName:0676b292-a5ce-4c78-807a-7f1ff712488e nodeName:}" failed. No retries permitted until 2026-10-31 14:55:39.691810459 +0000 UTC m=+580.418091420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls") pod "maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" (UID: "0676b292-a5ce-4c78-807a-7f1ff712488e") : secret "maas-api-e2e-migrate-b1c2330c-serving-cert" not found
Oct 31 14:55:39.192079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.192060    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.193994 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.193972    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.256679 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.250963    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdt9\" (UniqueName: \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.695700 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.695665    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.698565 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.698535    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") pod \"maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:39.913899 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:39.913870    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:40.292779 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:40.292756    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:55:40.294355 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:55:40.294325    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0676b292_a5ce_4c78_807a_7f1ff712488e.slice/crio-cb9b9ca13454b9de870b614629c306ea922206b6b608201a36b5bb35709c5703 WatchSource:0}: Error finding container cb9b9ca13454b9de870b614629c306ea922206b6b608201a36b5bb35709c5703: Status 404 returned error can't find the container with id cb9b9ca13454b9de870b614629c306ea922206b6b608201a36b5bb35709c5703
Oct 31 14:55:41.250464 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:41.250431    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" event={"ID":"0676b292-a5ce-4c78-807a-7f1ff712488e","Type":"ContainerStarted","Data":"cb9b9ca13454b9de870b614629c306ea922206b6b608201a36b5bb35709c5703"}
Oct 31 14:55:42.258065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:42.258026    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" event={"ID":"0676b292-a5ce-4c78-807a-7f1ff712488e","Type":"ContainerStarted","Data":"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"}
Oct 31 14:55:42.258464 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:42.258190    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:42.377018 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:42.376966    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" podStartSLOduration=2.536282291 podStartE2EDuration="4.376950532s" totalImagesPullingTime="1.840668241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:42.374876505 +0000 UTC m=+583.101157496" watchObservedRunningTime="2026-10-31 14:55:42.376950532 +0000 UTC m=+583.103231510"
Oct 31 14:55:45.685720 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.685687    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:55:45.689397 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.689380    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.745675 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.745643    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:55:45.755967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.755935    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447zr\" (UniqueName: \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.756126 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.755986    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.792108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.792072    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ee109dd4-dockercfg-p9nr9\""
Oct 31 14:55:45.792108 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.792072    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ee109dd4\""
Oct 31 14:55:45.856901 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.856871    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.857115 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.857011    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447zr\" (UniqueName: \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.857636 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.857617    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:45.919971 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:45.919933    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447zr\" (UniqueName: \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\") pod \"payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:46.002901 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:46.002793    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:46.378833 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:46.378810    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:55:46.380721 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:55:46.380694    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912c8c2d_3c4f_4012_8046_016a966eaf35.slice/crio-6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b WatchSource:0}: Error finding container 6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b: Status 404 returned error can't find the container with id 6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b
Oct 31 14:55:47.287719 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:47.287681    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" event={"ID":"912c8c2d-3c4f-4012-8046-016a966eaf35","Type":"ContainerStarted","Data":"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"}
Oct 31 14:55:47.288121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:47.287742    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" event={"ID":"912c8c2d-3c4f-4012-8046-016a966eaf35","Type":"ContainerStarted","Data":"6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b"}
Oct 31 14:55:47.288121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:47.287763    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:47.404876 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:47.404824    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" podStartSLOduration=2.404808819 podStartE2EDuration="2.404808819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:47.40189415 +0000 UTC m=+588.128175128" watchObservedRunningTime="2026-10-31 14:55:47.404808819 +0000 UTC m=+588.131089797"
Oct 31 14:55:48.233280 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:48.233248    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:55:48.269860 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:48.269829    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:55:49.238037 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:49.238008    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:55:58.296379 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:58.296351    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:55:59.471954 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:59.471927    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:55:59.483453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:55:59.483423    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 14:56:07.012106 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:07.012065    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:56:11.168419 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.168383    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"]
Oct 31 14:56:11.175122 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.175102    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.188996 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.188970    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ws7m\" (UniqueName: \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.189107 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.189091    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.189150 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.189127    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.227885 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.227846    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"]
Oct 31 14:56:11.274494 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.274462    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:56:11.274494 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.274471    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 14:56:11.289888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.289852    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.290091 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.289941    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ws7m\" (UniqueName: \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.290091 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.290037    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.290804 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.290783    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.292326 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.292306    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.348230 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.348187    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ws7m\" (UniqueName: \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\") pod \"maas-api-revoke-keys-e2e-ait-b671479d-hjp94\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.487951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.487854    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:11.863593 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:11.863550    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"]
Oct 31 14:56:11.865139 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:56:11.865110    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fd58c5_7176_4827_84a4_55f23b3a9c41.slice/crio-e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea WatchSource:0}: Error finding container e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea: Status 404 returned error can't find the container with id e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea
Oct 31 14:56:12.439188 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:12.439153    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94" event={"ID":"80fd58c5-7176-4827-84a4-55f23b3a9c41","Type":"ContainerStarted","Data":"e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea"}
Oct 31 14:56:15.461700 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:15.461663    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94" event={"ID":"80fd58c5-7176-4827-84a4-55f23b3a9c41","Type":"ContainerStarted","Data":"18d534ecc2fc3a422b2a8c1063ac97e86054e56d34122c47f4b19845b573741b"}
Oct 31 14:56:15.575495 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:15.575381    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94" podStartSLOduration=1.408071919 podStartE2EDuration="4.575366431s" totalImagesPullingTime="3.167294512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:15.574770685 +0000 UTC m=+616.301051676" watchObservedRunningTime="2026-10-31 14:56:15.575366431 +0000 UTC m=+616.301647419"
Oct 31 14:56:16.469155 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:16.469124    4549 generic.go:356] "Generic (PLEG): container finished" podID="80fd58c5-7176-4827-84a4-55f23b3a9c41" containerID="18d534ecc2fc3a422b2a8c1063ac97e86054e56d34122c47f4b19845b573741b" exitCode=0
Oct 31 14:56:16.469596 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:16.469184    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94" event={"ID":"80fd58c5-7176-4827-84a4-55f23b3a9c41","Type":"ContainerDied","Data":"18d534ecc2fc3a422b2a8c1063ac97e86054e56d34122c47f4b19845b573741b"}
Oct 31 14:56:17.611238 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.611215    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:17.656057 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.656023    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\") pod \"80fd58c5-7176-4827-84a4-55f23b3a9c41\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") "
Oct 31 14:56:17.656243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.656066    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\" (UniqueName: \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\") pod \"80fd58c5-7176-4827-84a4-55f23b3a9c41\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") "
Oct 31 14:56:17.656243 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.656158    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\") pod \"80fd58c5-7176-4827-84a4-55f23b3a9c41\" (UID: \"80fd58c5-7176-4827-84a4-55f23b3a9c41\") "
Oct 31 14:56:17.656420 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.656395    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca" pod "80fd58c5-7176-4827-84a4-55f23b3a9c41" (UID: "80fd58c5-7176-4827-84a4-55f23b3a9c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:17.658446 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.658422    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca" pod "80fd58c5-7176-4827-84a4-55f23b3a9c41" (UID: "80fd58c5-7176-4827-84a4-55f23b3a9c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:56:17.658536 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.658517    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m" pod "80fd58c5-7176-4827-84a4-55f23b3a9c41" (UID: "80fd58c5-7176-4827-84a4-55f23b3a9c41"). InnerVolumeSpecName "kube-api-access-6ws7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:17.756951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.756868    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-openshift-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:17.756951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.756896    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ws7m\" (UniqueName: \"kubernetes.io/projected/80fd58c5-7176-4827-84a4-55f23b3a9c41-kube-api-access-6ws7m\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:17.756951 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:17.756909    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80fd58c5-7176-4827-84a4-55f23b3a9c41-maas-api-xks-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:18.480646 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:18.480610    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94" event={"ID":"80fd58c5-7176-4827-84a4-55f23b3a9c41","Type":"ContainerDied","Data":"e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea"}
Oct 31 14:56:18.480821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:18.480667    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78e60e50934d2fa6457569eb079d058a495d20bbb397a17bcef4084b673d1ea"
Oct 31 14:56:18.480821 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:18.480621    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"
Oct 31 14:56:19.642257 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:19.642225    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:56:22.249176 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:22.249141    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:56:24.642960 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:24.642918    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" podUID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" containerName="payload-pre-processing" containerID="cri-o://df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a" gracePeriod=25
Oct 31 14:56:24.897022 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:24.896968    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:56:25.023078 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.023048    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\") pod \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") "
Oct 31 14:56:25.023258 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.023087    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\" (UniqueName: \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\") pod \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\" (UID: \"dab58c94-fd19-48fe-a5f8-892c4050bdd1\") "
Oct 31 14:56:25.023494 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.023470    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume" pod "dab58c94-fd19-48fe-a5f8-892c4050bdd1" (UID: "dab58c94-fd19-48fe-a5f8-892c4050bdd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:25.025540 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.025516    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf" pod "dab58c94-fd19-48fe-a5f8-892c4050bdd1" (UID: "dab58c94-fd19-48fe-a5f8-892c4050bdd1"). InnerVolumeSpecName "kube-api-access-rqgcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:25.124014 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.123983    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab58c94-fd19-48fe-a5f8-892c4050bdd1-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:25.124014 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.124008    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgcf\" (UniqueName: \"kubernetes.io/projected/dab58c94-fd19-48fe-a5f8-892c4050bdd1-kube-api-access-rqgcf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:25.518453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.518418    4549 generic.go:356] "Generic (PLEG): container finished" podID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" containerID="df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a" exitCode=0
Oct 31 14:56:25.518663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.518459    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"
Oct 31 14:56:25.518663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.518493    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" event={"ID":"dab58c94-fd19-48fe-a5f8-892c4050bdd1","Type":"ContainerDied","Data":"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"}
Oct 31 14:56:25.518663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.518541    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn" event={"ID":"dab58c94-fd19-48fe-a5f8-892c4050bdd1","Type":"ContainerDied","Data":"c3926928954cdd351dd0b044d89146aeb78bd52b678cc441feb5a52a7d08043e"}
Oct 31 14:56:25.518663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.518563    4549 scope.go:122] "RemoveContainer" containerID="df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"
Oct 31 14:56:25.527633 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.527616    4549 scope.go:122] "RemoveContainer" containerID="df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"
Oct 31 14:56:25.527943 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:25.527921    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a\": container with ID starting with df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a not found: ID does not exist" containerID="df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"
Oct 31 14:56:25.527991 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.527952    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a"} err="failed to get container status \"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a\": rpc error: code = NotFound desc = could not find container \"df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a\": container with ID starting with df588e2674d6940024243aa69b59f7b84d8f672d5a22cd8023e39c6e9e3fed5a not found: ID does not exist"
Oct 31 14:56:25.720283 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.720199    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:56:25.723183 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:25.723160    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33bc513e-68c6bc58b7-c44sn"]
Oct 31 14:56:27.250367 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.250308    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" podUID="2b173150-d48c-4360-8372-3ee84883e994" containerName="payload-pre-processing" containerID="cri-o://7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f" gracePeriod=25
Oct 31 14:56:27.422859 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.422825    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" path="/var/lib/kubelet/pods/dab58c94-fd19-48fe-a5f8-892c4050bdd1/volumes"
Oct 31 14:56:27.529809 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.529782    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:56:27.534042 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.534011    4549 generic.go:356] "Generic (PLEG): container finished" podID="2b173150-d48c-4360-8372-3ee84883e994" containerID="7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f" exitCode=0
Oct 31 14:56:27.534168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.534056    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"
Oct 31 14:56:27.534168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.534070    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" event={"ID":"2b173150-d48c-4360-8372-3ee84883e994","Type":"ContainerDied","Data":"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"}
Oct 31 14:56:27.534168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.534122    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg" event={"ID":"2b173150-d48c-4360-8372-3ee84883e994","Type":"ContainerDied","Data":"228e09d6e0558b4267b4db8925779202a4e13a9d2969cbaa3e62aaf554661a1c"}
Oct 31 14:56:27.534168 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.534151    4549 scope.go:122] "RemoveContainer" containerID="7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"
Oct 31 14:56:27.545064 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.545036    4549 scope.go:122] "RemoveContainer" containerID="7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"
Oct 31 14:56:27.545349 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:27.545332    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f\": container with ID starting with 7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f not found: ID does not exist" containerID="7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"
Oct 31 14:56:27.545410 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.545360    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f"} err="failed to get container status \"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f\": rpc error: code = NotFound desc = could not find container \"7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f\": container with ID starting with 7a50e6b48cd898b0662af4aa564a692859c534ba2ff5e7b32f728f9810bb971f not found: ID does not exist"
Oct 31 14:56:27.652900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.651975    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\") pod \"2b173150-d48c-4360-8372-3ee84883e994\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") "
Oct 31 14:56:27.652900 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.652025    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\" (UniqueName: \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\") pod \"2b173150-d48c-4360-8372-3ee84883e994\" (UID: \"2b173150-d48c-4360-8372-3ee84883e994\") "
Oct 31 14:56:27.653184 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.653149    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume" pod "2b173150-d48c-4360-8372-3ee84883e994" (UID: "2b173150-d48c-4360-8372-3ee84883e994"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:27.655747 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.655718    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c" pod "2b173150-d48c-4360-8372-3ee84883e994" (UID: "2b173150-d48c-4360-8372-3ee84883e994"). InnerVolumeSpecName "kube-api-access-wqj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:27.753078 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.753039    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b173150-d48c-4360-8372-3ee84883e994-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:27.753078 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.753077    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqj2c\" (UniqueName: \"kubernetes.io/projected/2b173150-d48c-4360-8372-3ee84883e994-kube-api-access-wqj2c\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:27.952648 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.952617    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:56:27.956730 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:27.956707    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b671479d-59584f4c79-jbfzg"]
Oct 31 14:56:29.419293 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:29.419259    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b173150-d48c-4360-8372-3ee84883e994" path="/var/lib/kubelet/pods/2b173150-d48c-4360-8372-3ee84883e994/volumes"
Oct 31 14:56:29.789628 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:29.789590    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:56:29.789870 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:29.789846    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" podUID="0676b292-a5ce-4c78-807a-7f1ff712488e" containerName="maas-api" containerID="cri-o://433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9" gracePeriod=30
Oct 31 14:56:30.073003 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.072975    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:56:30.178136 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.178101    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\") pod \"0676b292-a5ce-4c78-807a-7f1ff712488e\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") "
Oct 31 14:56:30.178334 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.178150    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\" (UniqueName: \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\") pod \"0676b292-a5ce-4c78-807a-7f1ff712488e\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") "
Oct 31 14:56:30.178334 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.178197    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") pod \"0676b292-a5ce-4c78-807a-7f1ff712488e\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") "
Oct 31 14:56:30.178334 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.178248    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\") pod \"0676b292-a5ce-4c78-807a-7f1ff712488e\" (UID: \"0676b292-a5ce-4c78-807a-7f1ff712488e\") "
Oct 31 14:56:30.178558 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.178535    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca" pod "0676b292-a5ce-4c78-807a-7f1ff712488e" (UID: "0676b292-a5ce-4c78-807a-7f1ff712488e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:30.180635 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.180607    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9" pod "0676b292-a5ce-4c78-807a-7f1ff712488e" (UID: "0676b292-a5ce-4c78-807a-7f1ff712488e"). InnerVolumeSpecName "kube-api-access-xjdt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:30.180635 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.180615    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs" pod "0676b292-a5ce-4c78-807a-7f1ff712488e" (UID: "0676b292-a5ce-4c78-807a-7f1ff712488e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:56:30.180768 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.180605    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls" pod "0676b292-a5ce-4c78-807a-7f1ff712488e" (UID: "0676b292-a5ce-4c78-807a-7f1ff712488e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:56:30.279221 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.279187    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:30.279221 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.279220    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0676b292-a5ce-4c78-807a-7f1ff712488e-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:30.279433 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.279237    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdt9\" (UniqueName: \"kubernetes.io/projected/0676b292-a5ce-4c78-807a-7f1ff712488e-kube-api-access-xjdt9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:30.279433 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.279252    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0676b292-a5ce-4c78-807a-7f1ff712488e-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:30.288038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288005    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"]
Oct 31 14:56:30.288551 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288504    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" containerName="payload-pre-processing"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288554    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" containerName="payload-pre-processing"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288590    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0676b292-a5ce-4c78-807a-7f1ff712488e" containerName="maas-api"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288600    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0676b292-a5ce-4c78-807a-7f1ff712488e" containerName="maas-api"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288647    4549 cpu_manager.go:414] "containerMap: removing container" podUID="2b173150-d48c-4360-8372-3ee84883e994" containerName="payload-pre-processing"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288655    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b173150-d48c-4360-8372-3ee84883e994" containerName="payload-pre-processing"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288664    4549 cpu_manager.go:414] "containerMap: removing container" podUID="80fd58c5-7176-4827-84a4-55f23b3a9c41" containerName="revoke-keys"
Oct 31 14:56:30.288712 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288670    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fd58c5-7176-4827-84a4-55f23b3a9c41" containerName="revoke-keys"
Oct 31 14:56:30.289038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288794    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab58c94-fd19-48fe-a5f8-892c4050bdd1" containerName="payload-pre-processing"
Oct 31 14:56:30.289038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288809    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0676b292-a5ce-4c78-807a-7f1ff712488e" containerName="maas-api"
Oct 31 14:56:30.289038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288824    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b173150-d48c-4360-8372-3ee84883e994" containerName="payload-pre-processing"
Oct 31 14:56:30.289038 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.288832    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fd58c5-7176-4827-84a4-55f23b3a9c41" containerName="revoke-keys"
Oct 31 14:56:30.292552 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.292530    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.357412 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.353452    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"]
Oct 31 14:56:30.357412 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.353765    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 14:56:30.357412 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.353826    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 14:56:30.379906 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.379870    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lwp\" (UniqueName: \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.380065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.379919    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.380065 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.379945    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.481249 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.481218    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.481757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.481263    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.481757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.481345    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lwp\" (UniqueName: \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.482062 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.482040    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.483718 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.483693    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.538496 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.538462    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lwp\" (UniqueName: \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\") pod \"maas-api-revoke-keys-e2e-ait-08b12cc9-lk547\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.557299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.557219    4549 generic.go:356] "Generic (PLEG): container finished" podID="0676b292-a5ce-4c78-807a-7f1ff712488e" containerID="433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9" exitCode=0
Oct 31 14:56:30.557299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.557260    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"
Oct 31 14:56:30.557299 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.557298    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" event={"ID":"0676b292-a5ce-4c78-807a-7f1ff712488e","Type":"ContainerDied","Data":"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"}
Oct 31 14:56:30.557549 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.557366    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr" event={"ID":"0676b292-a5ce-4c78-807a-7f1ff712488e","Type":"ContainerDied","Data":"cb9b9ca13454b9de870b614629c306ea922206b6b608201a36b5bb35709c5703"}
Oct 31 14:56:30.557549 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.557397    4549 scope.go:122] "RemoveContainer" containerID="433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"
Oct 31 14:56:30.567905 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.567883    4549 scope.go:122] "RemoveContainer" containerID="433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"
Oct 31 14:56:30.568209 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:30.568184    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9\": container with ID starting with 433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9 not found: ID does not exist" containerID="433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"
Oct 31 14:56:30.568265 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.568223    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9"} err="failed to get container status \"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9\": rpc error: code = NotFound desc = could not find container \"433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9\": container with ID starting with 433e4e19971483c417561de4b5916357df5fec5528e5e907d63e7327b63e79f9 not found: ID does not exist"
Oct 31 14:56:30.604948 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.604917    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:30.726847 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.726817    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:56:30.730229 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.730199    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b1c2330c-7f86cf7598-94ltr"]
Oct 31 14:56:30.977415 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:30.977384    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"]
Oct 31 14:56:30.978853 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:56:30.978819    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684e3734_7634_4304_afc9_09d941e99cbe.slice/crio-4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d WatchSource:0}: Error finding container 4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d: Status 404 returned error can't find the container with id 4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d
Oct 31 14:56:31.233940 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.233909    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 14:56:31.237560 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.237540    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.289580 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.289529    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-029d26-openshift-default-dockercfg-lxm75\""
Oct 31 14:56:31.297815 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.297788    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 14:56:31.394467 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394437    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394485    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394562    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394612    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394628    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394695    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6pl\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394867 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394842    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394909 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394887    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.394958 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.394943    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.417498 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.417471    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0676b292-a5ce-4c78-807a-7f1ff712488e" path="/var/lib/kubelet/pods/0676b292-a5ce-4c78-807a-7f1ff712488e/volumes"
Oct 31 14:56:31.496079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.495983    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496021    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496037    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496079 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496082    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496110    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496137    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496171    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496195    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496221    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6pl\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496553    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.496760 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.496649    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.497084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.497003    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.497084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.497002    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.497084 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.497055    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.498489 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.498469    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.498945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.498926    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.556763 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.556734    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.557027 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.557007    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6pl\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.566466 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.566178    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547" event={"ID":"684e3734-7634-4304-afc9-09d941e99cbe","Type":"ContainerStarted","Data":"91a0d55784c9b9b95e504635aab249aa0c72914d3ed4fa6372066825e16e64ed"}
Oct 31 14:56:31.566466 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.566235    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547" event={"ID":"684e3734-7634-4304-afc9-09d941e99cbe","Type":"ContainerStarted","Data":"4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d"}
Oct 31 14:56:31.681008 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.680950    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547" podStartSLOduration=1.6809349409999998 podStartE2EDuration="1.680934941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:31.67902862 +0000 UTC m=+632.405309609" watchObservedRunningTime="2026-10-31 14:56:31.680934941 +0000 UTC m=+632.407215919"
Oct 31 14:56:31.851768 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.851739    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:31.861729 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.861704    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"]
Oct 31 14:56:31.864131 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:31.864075    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b671479d-hjp94"]
Oct 31 14:56:32.280469 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.280447    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 14:56:32.282482 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:56:32.282451    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbb75fd_257d_4afa_9673_e8282b3da468.slice/crio-39b2c528dc69539b71a0eef36ec69fbced46853051dc5a5479bacd707315d56e WatchSource:0}: Error finding container 39b2c528dc69539b71a0eef36ec69fbced46853051dc5a5479bacd707315d56e: Status 404 returned error can't find the container with id 39b2c528dc69539b71a0eef36ec69fbced46853051dc5a5479bacd707315d56e
Oct 31 14:56:32.285326 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.285291    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:56:32.285424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.285369    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:56:32.285424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.285417    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:56:32.575126 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.575015    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerStarted","Data":"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"}
Oct 31 14:56:32.575126 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.575089    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerStarted","Data":"39b2c528dc69539b71a0eef36ec69fbced46853051dc5a5479bacd707315d56e"}
Oct 31 14:56:32.576942 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.576912    4549 generic.go:356] "Generic (PLEG): container finished" podID="684e3734-7634-4304-afc9-09d941e99cbe" containerID="91a0d55784c9b9b95e504635aab249aa0c72914d3ed4fa6372066825e16e64ed" exitCode=0
Oct 31 14:56:32.577086 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.577003    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547" event={"ID":"684e3734-7634-4304-afc9-09d941e99cbe","Type":"ContainerDied","Data":"91a0d55784c9b9b95e504635aab249aa0c72914d3ed4fa6372066825e16e64ed"}
Oct 31 14:56:32.714285 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.714234    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podStartSLOduration=1.714220117 podStartE2EDuration="1.714220117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:32.712987857 +0000 UTC m=+633.439268858" watchObservedRunningTime="2026-10-31 14:56:32.714220117 +0000 UTC m=+633.440501095"
Oct 31 14:56:32.852664 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:32.852558    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:33.418325 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.418298    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fd58c5-7176-4827-84a4-55f23b3a9c41" path="/var/lib/kubelet/pods/80fd58c5-7176-4827-84a4-55f23b3a9c41/volumes"
Oct 31 14:56:33.732728 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.732703    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:33.818839 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.818792    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\" (UniqueName: \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\") pod \"684e3734-7634-4304-afc9-09d941e99cbe\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") "
Oct 31 14:56:33.819035 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.818855    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\") pod \"684e3734-7634-4304-afc9-09d941e99cbe\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") "
Oct 31 14:56:33.819035 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.818913    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\") pod \"684e3734-7634-4304-afc9-09d941e99cbe\" (UID: \"684e3734-7634-4304-afc9-09d941e99cbe\") "
Oct 31 14:56:33.819659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.819629    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca" pod "684e3734-7634-4304-afc9-09d941e99cbe" (UID: "684e3734-7634-4304-afc9-09d941e99cbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:33.821844 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.821819    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp" pod "684e3734-7634-4304-afc9-09d941e99cbe" (UID: "684e3734-7634-4304-afc9-09d941e99cbe"). InnerVolumeSpecName "kube-api-access-g8lwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:33.821945 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.821860    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca" pod "684e3734-7634-4304-afc9-09d941e99cbe" (UID: "684e3734-7634-4304-afc9-09d941e99cbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 14:56:33.858278 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.858249    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:33.920447 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.920414    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8lwp\" (UniqueName: \"kubernetes.io/projected/684e3734-7634-4304-afc9-09d941e99cbe-kube-api-access-g8lwp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:33.920447 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.920442    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-openshift-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:33.920697 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:33.920458    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684e3734-7634-4304-afc9-09d941e99cbe-maas-api-xks-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:34.592973 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:34.592925    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547" event={"ID":"684e3734-7634-4304-afc9-09d941e99cbe","Type":"ContainerDied","Data":"4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d"}
Oct 31 14:56:34.593190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:34.593002    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4479d91557ff806a5c320654d3ea2f1c0b883014d4cf8a4552f0efe282978f8d"
Oct 31 14:56:34.593190 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:34.592950    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"
Oct 31 14:56:34.593552 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:34.593531    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:34.594535 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:34.594518    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:56:37.011325 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:37.011289    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:56:45.311670 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:45.311627    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:56:50.312659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.312603    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" podUID="912c8c2d-3c4f-4012-8046-016a966eaf35" containerName="payload-pre-processing" containerID="cri-o://63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892" gracePeriod=25
Oct 31 14:56:50.555453 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.555422    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:56:50.587629 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.587533    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\" (UniqueName: \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\") pod \"912c8c2d-3c4f-4012-8046-016a966eaf35\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") "
Oct 31 14:56:50.589845 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.589816    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr" pod "912c8c2d-3c4f-4012-8046-016a966eaf35" (UID: "912c8c2d-3c4f-4012-8046-016a966eaf35"). InnerVolumeSpecName "kube-api-access-447zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 14:56:50.683386 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.683354    4549 generic.go:356] "Generic (PLEG): container finished" podID="912c8c2d-3c4f-4012-8046-016a966eaf35" containerID="63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892" exitCode=0
Oct 31 14:56:50.683614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.683394    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"
Oct 31 14:56:50.683614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.683432    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" event={"ID":"912c8c2d-3c4f-4012-8046-016a966eaf35","Type":"ContainerDied","Data":"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"}
Oct 31 14:56:50.683614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.683488    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6" event={"ID":"912c8c2d-3c4f-4012-8046-016a966eaf35","Type":"ContainerDied","Data":"6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b"}
Oct 31 14:56:50.683614 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.683510    4549 scope.go:122] "RemoveContainer" containerID="63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"
Oct 31 14:56:50.688833 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.688810    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\") pod \"912c8c2d-3c4f-4012-8046-016a966eaf35\" (UID: \"912c8c2d-3c4f-4012-8046-016a966eaf35\") "
Oct 31 14:56:50.689141 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.689121    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447zr\" (UniqueName: \"kubernetes.io/projected/912c8c2d-3c4f-4012-8046-016a966eaf35-kube-api-access-447zr\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:50.689218 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.689173    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume" pod "912c8c2d-3c4f-4012-8046-016a966eaf35" (UID: "912c8c2d-3c4f-4012-8046-016a966eaf35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 14:56:50.692896 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.692879    4549 scope.go:122] "RemoveContainer" containerID="63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"
Oct 31 14:56:50.693185 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:50.693162    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892\": container with ID starting with 63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892 not found: ID does not exist" containerID="63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"
Oct 31 14:56:50.693272 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.693193    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892"} err="failed to get container status \"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892\": rpc error: code = NotFound desc = could not find container \"63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892\": container with ID starting with 63b05ee1464891c7f7913cbf34609c2ec722e68361b6dc66a701a0045db90892 not found: ID does not exist"
Oct 31 14:56:50.790019 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:50.789984    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/912c8c2d-3c4f-4012-8046-016a966eaf35-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 14:56:51.011508 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:51.011467    4549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912c8c2d_3c4f_4012_8046_016a966eaf35.slice/crio-6b64a07ee9f2c201e8b24bc9b5c8b23f5725a7b74edb06fc632b026fe56cd10b\": RecentStats: unable to find data in memory cache]"
Oct 31 14:56:51.103547 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.103514    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:56:51.105771 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.105750    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ee109dd4-776dd97755-sdrt6"]
Oct 31 14:56:51.418068 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.418041    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912c8c2d-3c4f-4012-8046-016a966eaf35" path="/var/lib/kubelet/pods/912c8c2d-3c4f-4012-8046-016a966eaf35/volumes"
Oct 31 14:56:51.967765 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.967730    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 14:56:51.968788 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968761    4549 cpu_manager.go:414] "containerMap: removing container" podUID="912c8c2d-3c4f-4012-8046-016a966eaf35" containerName="payload-pre-processing"
Oct 31 14:56:51.968788 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968782    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912c8c2d-3c4f-4012-8046-016a966eaf35" containerName="payload-pre-processing"
Oct 31 14:56:51.968953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968802    4549 cpu_manager.go:414] "containerMap: removing container" podUID="684e3734-7634-4304-afc9-09d941e99cbe" containerName="revoke-keys"
Oct 31 14:56:51.968953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968808    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684e3734-7634-4304-afc9-09d941e99cbe" containerName="revoke-keys"
Oct 31 14:56:51.968953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968875    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="684e3734-7634-4304-afc9-09d941e99cbe" containerName="revoke-keys"
Oct 31 14:56:51.968953 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.968888    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="912c8c2d-3c4f-4012-8046-016a966eaf35" containerName="payload-pre-processing"
Oct 31 14:56:51.973663 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:51.973645    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.002101 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.002071    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.002291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.002214    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.002291 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.002240    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.002395 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.002370    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlmp\" (UniqueName: \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.029049 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.029012    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 14:56:52.075816 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.075782    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-029d26-serving-cert\""
Oct 31 14:56:52.076006 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.075820    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 14:56:52.076006 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.075889    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 14:56:52.077662 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.077647    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 14:56:52.077662 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.077648    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 14:56:52.103688 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.103657    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.103837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.103783    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.103837 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.103808    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.103960 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:52.103829    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-serving-cert: secret "maas-api-e2e-worker-w2-029d26-serving-cert" not found
Oct 31 14:56:52.103960 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.103886    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlmp\" (UniqueName: \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.103960 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:56:52.103915    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls podName:1aa8b25f-3105-4fbd-aea0-060e3de136fd nodeName:}" failed. No retries permitted until 2026-10-31 14:56:52.603890406 +0000 UTC m=+653.330171378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls") pod "maas-api-e2e-worker-w2-029d26-9c758649-px7nl" (UID: "1aa8b25f-3105-4fbd-aea0-060e3de136fd") : secret "maas-api-e2e-worker-w2-029d26-serving-cert" not found
Oct 31 14:56:52.104721 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.104692    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.106827 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.106800    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.160480 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.160442    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlmp\" (UniqueName: \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.608215 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.608184    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.610722 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.610704    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") pod \"maas-api-e2e-worker-w2-029d26-9c758649-px7nl\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:52.886671 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:52.886588    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:53.258959 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:53.258934    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 14:56:53.260416 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:56:53.260381    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa8b25f_3105_4fbd_aea0_060e3de136fd.slice/crio-c73912a333668cdc692bb2317ca6fe1f4086466be92944dbcc587bb7e6b2fc92 WatchSource:0}: Error finding container c73912a333668cdc692bb2317ca6fe1f4086466be92944dbcc587bb7e6b2fc92: Status 404 returned error can't find the container with id c73912a333668cdc692bb2317ca6fe1f4086466be92944dbcc587bb7e6b2fc92
Oct 31 14:56:53.261754 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:53.261736    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 14:56:53.702424 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:53.702388    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" event={"ID":"1aa8b25f-3105-4fbd-aea0-060e3de136fd","Type":"ContainerStarted","Data":"c73912a333668cdc692bb2317ca6fe1f4086466be92944dbcc587bb7e6b2fc92"}
Oct 31 14:56:54.710058 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:54.710009    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" event={"ID":"1aa8b25f-3105-4fbd-aea0-060e3de136fd","Type":"ContainerStarted","Data":"54f32d506712b1f248f04a1922c0794c52bdc29da7972b05472dfe54e53464e7"}
Oct 31 14:56:54.710643 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:54.710170    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:56:54.826538 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:56:54.826474    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" podStartSLOduration=2.820619243 podStartE2EDuration="3.826453482s" totalImagesPullingTime="1.005834239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:54.824046823 +0000 UTC m=+655.550327801" watchObservedRunningTime="2026-10-31 14:56:54.826453482 +0000 UTC m=+655.552734464"
Oct 31 14:57:00.724478 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:00.724444    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 14:57:03.322990 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:03.322952    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"]
Oct 31 14:57:03.353956 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:03.353922    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-08b12cc9-lk547"]
Oct 31 14:57:03.428177 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:03.428140    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684e3734-7634-4304-afc9-09d941e99cbe" path="/var/lib/kubelet/pods/684e3734-7634-4304-afc9-09d941e99cbe/volumes"
Oct 31 14:57:07.015919 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:57:07.015879    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:57:09.804156 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.804119    4549 generic.go:356] "Generic (PLEG): container finished" podID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerID="6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74" exitCode=0
Oct 31 14:57:09.804737 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.804194    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerDied","Data":"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"}
Oct 31 14:57:09.804737 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.804556    4549 scope.go:122] "RemoveContainer" containerID="6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"
Oct 31 14:57:09.805840 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.805809    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:57:09.805967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.805893    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:57:09.805967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:09.805943    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:57:10.834030 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:10.833903    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerStarted","Data":"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"}
Oct 31 14:57:10.852759 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:10.852729    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:57:10.852939 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:10.852920    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:10.852984 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:10.852965    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:11.852561 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:11.852523    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:11.853085 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:11.852614    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:12.853066 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:12.853028    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:12.853459 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:12.853077    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:13.852288 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:13.852257    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:13.852458 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:13.852306    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:14.852105 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:14.852077    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:14.852604 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:14.852129    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:15.852418 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:15.852374    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 14:57:15.852872 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:15.852449    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 14:57:16.856644 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:16.856615    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:57:16.857228 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:16.857019    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:57:16.857864 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:16.857845    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 14:57:30.275075 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.275041    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 14:57:30.278879 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.278858    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.310967 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.310930    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.311141 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.310973    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlrd\" (UniqueName: \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.311213 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.311151    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.311213 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.311201    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.382817 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.382783    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-57815b-serving-cert\""
Oct 31 14:57:30.385276 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.385246    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 14:57:30.412121 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.412086    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.413693 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.413661    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.413840 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.413761    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.413840 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.413793    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlrd\" (UniqueName: \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.414603 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:57:30.414564    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-serving-cert: secret "maas-api-e2e-worker-w4-57815b-serving-cert" not found
Oct 31 14:57:30.414825 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:57:30.414805    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls podName:4814f89f-7d1d-4c5c-bcbc-d3772636b7de nodeName:}" failed. No retries permitted until 2026-10-31 14:57:30.914783905 +0000 UTC m=+691.641064880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls") pod "maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" (UID: "4814f89f-7d1d-4c5c-bcbc-d3772636b7de") : secret "maas-api-e2e-worker-w4-57815b-serving-cert" not found
Oct 31 14:57:30.414954 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.414853    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.417194 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.417166    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.475848 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.473857    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlrd\" (UniqueName: \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.917776 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.917744    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:30.920317 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:30.920292    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") pod \"maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:31.192124 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:31.192043    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:31.564085 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:31.564060    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 14:57:31.565742 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:57:31.565716    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4814f89f_7d1d_4c5c_bcbc_d3772636b7de.slice/crio-da890331b3ad4490ed7c7a2f84960b419edcbdedbb0bbb7e22e9aef35ffb3abd WatchSource:0}: Error finding container da890331b3ad4490ed7c7a2f84960b419edcbdedbb0bbb7e22e9aef35ffb3abd: Status 404 returned error can't find the container with id da890331b3ad4490ed7c7a2f84960b419edcbdedbb0bbb7e22e9aef35ffb3abd
Oct 31 14:57:31.959021 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:31.958939    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" event={"ID":"4814f89f-7d1d-4c5c-bcbc-d3772636b7de","Type":"ContainerStarted","Data":"da890331b3ad4490ed7c7a2f84960b419edcbdedbb0bbb7e22e9aef35ffb3abd"}
Oct 31 14:57:32.968232 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:32.968195    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" event={"ID":"4814f89f-7d1d-4c5c-bcbc-d3772636b7de","Type":"ContainerStarted","Data":"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"}
Oct 31 14:57:32.968659 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:32.968266    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:33.102869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:33.102803    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" podStartSLOduration=2.29122464 podStartE2EDuration="3.102781644s" totalImagesPullingTime="811.557004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:33.099640998 +0000 UTC m=+693.825921977" watchObservedRunningTime="2026-10-31 14:57:33.102781644 +0000 UTC m=+693.829062627"
Oct 31 14:57:37.011633 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:57:37.011594    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:57:38.980524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:38.980497    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 14:57:40.671860 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.671824    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 14:57:40.676211 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.676186    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.779783 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.779745    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-fb2348-serving-cert\""
Oct 31 14:57:40.782228 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.782206    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 14:57:40.821016 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.820983    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.821209 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.821055    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.821273 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.821225    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.821330 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.821275    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmz42\" (UniqueName: \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.922794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.922691    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.922794 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.922781    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.923039 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.922959    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmz42\" (UniqueName: \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.923101 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.923041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.923630 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.923541    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.925440 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.925414    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.925689 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.925669    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.980029 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.980001    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmz42\" (UniqueName: \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\") pod \"maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:40.991362 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:40.991337    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:41.363223 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:41.363192    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 14:57:41.364626 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:57:41.364595    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92463f88_bab2_4936_9c20_350c03b92071.slice/crio-f90242350d2ed6beb2a46265a404d5c42334c5657211c3e7449579e265c897a9 WatchSource:0}: Error finding container f90242350d2ed6beb2a46265a404d5c42334c5657211c3e7449579e265c897a9: Status 404 returned error can't find the container with id f90242350d2ed6beb2a46265a404d5c42334c5657211c3e7449579e265c897a9
Oct 31 14:57:42.022608 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:42.022549    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" event={"ID":"92463f88-bab2-4936-9c20-350c03b92071","Type":"ContainerStarted","Data":"f90242350d2ed6beb2a46265a404d5c42334c5657211c3e7449579e265c897a9"}
Oct 31 14:57:43.031276 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:43.031246    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" event={"ID":"92463f88-bab2-4936-9c20-350c03b92071","Type":"ContainerStarted","Data":"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"}
Oct 31 14:57:43.031735 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:43.031318    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:57:43.146153 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:43.146095    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" podStartSLOduration=2.333456222 podStartE2EDuration="3.146082324s" totalImagesPullingTime="812.626102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:43.145471775 +0000 UTC m=+703.871752752" watchObservedRunningTime="2026-10-31 14:57:43.146082324 +0000 UTC m=+703.872363301"
Oct 31 14:57:49.042905 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:57:49.042874    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 14:58:07.012361 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:58:07.012241    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:58:30.768596 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.768546    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 14:58:30.772372 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.772348    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.827999 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.827966    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"kube-root-ca.crt\""
Oct 31 14:58:30.840667 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.840641    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"openshift-service-ca.crt\""
Oct 31 14:58:30.877203 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.877164    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"default-dockercfg-w4fm4\""
Oct 31 14:58:30.877361 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.877322    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"simulator-w4-57815b-kserve-self-signed-certs\""
Oct 31 14:58:30.879691 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.879671    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 14:58:30.922843 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.922812    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.923021 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.922933    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.923141 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.923112    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.923209 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.923164    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4vz\" (UniqueName: \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.923263 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.923233    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:30.923318 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:30.923259    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024719 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024633    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024719 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024694    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024722    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4vz\" (UniqueName: \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024773    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024795    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.024926 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.024826    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.025211 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.025150    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.025270 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.025174    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.025270 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.025239    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.027610 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.027564    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.027903 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.027884    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.081944 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.081909    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4vz\" (UniqueName: \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\") pod \"simulator-w4-57815b-kserve-5d447fb94c-r89zh\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") " pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.083763 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.083736    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:31.466521 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:31.466493    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 14:58:31.468160 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:58:31.468133    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf98a38f_d6c2_45ad_a43c_a533ed3b59be.slice/crio-cb3f6882a0ca60ca0c6b3572bbf364755e873736dc1b2d4d8cb9ebbf3867c8ca WatchSource:0}: Error finding container cb3f6882a0ca60ca0c6b3572bbf364755e873736dc1b2d4d8cb9ebbf3867c8ca: Status 404 returned error can't find the container with id cb3f6882a0ca60ca0c6b3572bbf364755e873736dc1b2d4d8cb9ebbf3867c8ca
Oct 31 14:58:32.324848 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:32.324814    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" event={"ID":"bf98a38f-d6c2-45ad-a43c-a533ed3b59be","Type":"ContainerStarted","Data":"cb3f6882a0ca60ca0c6b3572bbf364755e873736dc1b2d4d8cb9ebbf3867c8ca"}
Oct 31 14:58:34.339133 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:34.339101    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" event={"ID":"bf98a38f-d6c2-45ad-a43c-a533ed3b59be","Type":"ContainerStarted","Data":"63be2ff5a0e8f4f147e578fae690b4c58dd584cd60c280268d6d3901eade31ec"}
Oct 31 14:58:35.084524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:35.084484    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:37.014392 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:58:37.014349    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:58:41.084637 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:41.084539    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:41.096861 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:41.096839    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:58:41.211863 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:41.211811    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podStartSLOduration=9.089045335 podStartE2EDuration="11.211797303s" totalImagesPullingTime="2.122751968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:58:34.491920232 +0000 UTC m=+755.218201210" watchObservedRunningTime="2026-10-31 14:58:41.211797303 +0000 UTC m=+761.938078280"
Oct 31 14:58:41.393282 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:58:41.393205    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 14:59:07.011642 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:59:07.011596    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:59:23.741076 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.741042    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 14:59:23.745468 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.745444    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.800063 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.800033    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 14:59:23.816421 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816373    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lsj\" (UniqueName: \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.816645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816433    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.816645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816505    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.816645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816536    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.816645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816587    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.816645 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.816613    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.846862 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.846823    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"premium-w4-57815b-kserve-self-signed-certs\""
Oct 31 14:59:23.917444 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917409    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lsj\" (UniqueName: \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.917669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917470    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.917669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917505    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.917669 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917529    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.917850 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917660    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.917850 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.917700    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.918029 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.918002    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.918102 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.918034    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.918217 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.918197    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.920160 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.920133    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.920386 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.920368    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:23.974525 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:23.974480    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lsj\" (UniqueName: \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\") pod \"premium-w4-57815b-kserve-5d4f6694c4-k2vdq\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") " pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:24.057878 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:24.057842    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:24.433332 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:24.433304    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 14:59:24.435091 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:59:24.435064    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3357555d_1e57_458b_89d0_f4994654cfb0.slice/crio-229d6db40a7022d3a1b84d2aa18ae33272df3cb476f71256bc2e3781c83db9e1 WatchSource:0}: Error finding container 229d6db40a7022d3a1b84d2aa18ae33272df3cb476f71256bc2e3781c83db9e1: Status 404 returned error can't find the container with id 229d6db40a7022d3a1b84d2aa18ae33272df3cb476f71256bc2e3781c83db9e1
Oct 31 14:59:24.651611 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:24.651560    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" event={"ID":"3357555d-1e57-458b-89d0-f4994654cfb0","Type":"ContainerStarted","Data":"7c67f629e7303419331a6c34a293dca2f077b08d062837a263edf29fad244784"}
Oct 31 14:59:24.651814 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:24.651619    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" event={"ID":"3357555d-1e57-458b-89d0-f4994654cfb0","Type":"ContainerStarted","Data":"229d6db40a7022d3a1b84d2aa18ae33272df3cb476f71256bc2e3781c83db9e1"}
Oct 31 14:59:24.766869 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:24.766822    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podStartSLOduration=1.76680665 podStartE2EDuration="1.76680665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:24.765164586 +0000 UTC m=+805.491445561" watchObservedRunningTime="2026-10-31 14:59:24.76680665 +0000 UTC m=+805.493087627"
Oct 31 14:59:25.058638 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:25.058544    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:34.058546 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:34.058515    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:34.071295 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:34.071266    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:34.727487 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:34.727459    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 14:59:37.012138 ip-10-0-139-115 kubenswrapper[4549]: E1031 14:59:37.012101    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 14:59:52.438012 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.437974    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 14:59:52.441947 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.441921    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.496337 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.496201    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-57815b-openshift-default-dockercfg-9dqr9\""
Oct 31 14:59:52.548842 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.548807    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 14:59:52.594650 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594610    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.594650 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594654    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.594888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594733    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.594888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594758    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtxb\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.594888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594773    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.594888 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594795    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.595054 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.594972    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.595054 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.595018    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.595132 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.595105    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696076 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.695986    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696076 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696076 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696073    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696365 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696105    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696365 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696266    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696365 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696329    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696387    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696419    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtxb\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696449    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696508    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696767 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696723    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696824 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696766    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696892 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696869    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.696946 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.696909    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.698885 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.698863    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.699393 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.699371    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.753650 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.753618    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.753811 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.753792    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtxb\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:52.757524 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:52.757503    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:53.187632 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.187598    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 14:59:53.188405 ip-10-0-139-115 kubenswrapper[4549]: W1031 14:59:53.188374    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094370f9_3682_419e_b874_ccbd046aed7b.slice/crio-8c0fe56dbd631991606ce5b7fd2bf43589e376e16a4c6074c3f7d9ea545c15ea WatchSource:0}: Error finding container 8c0fe56dbd631991606ce5b7fd2bf43589e376e16a4c6074c3f7d9ea545c15ea: Status 404 returned error can't find the container with id 8c0fe56dbd631991606ce5b7fd2bf43589e376e16a4c6074c3f7d9ea545c15ea
Oct 31 14:59:53.190690 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.190655    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:59:53.190764 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.190720    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:59:53.190764 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.190748    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 14:59:53.865239 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.865192    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" event={"ID":"094370f9-3682-419e-b874-ccbd046aed7b","Type":"ContainerStarted","Data":"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"}
Oct 31 14:59:53.865757 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.865269    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" event={"ID":"094370f9-3682-419e-b874-ccbd046aed7b","Type":"ContainerStarted","Data":"8c0fe56dbd631991606ce5b7fd2bf43589e376e16a4c6074c3f7d9ea545c15ea"}
Oct 31 14:59:53.983152 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:53.983102    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podStartSLOduration=1.9830896249999999 podStartE2EDuration="1.983089625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:53.980771474 +0000 UTC m=+834.707052477" watchObservedRunningTime="2026-10-31 14:59:53.983089625 +0000 UTC m=+834.709370667"
Oct 31 14:59:54.757901 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:54.757863    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 14:59:54.759178 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:54.759150    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:54.759407 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:54.759382    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 14:59:55.758836 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:55.758801    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:55.759213 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:55.758856    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 14:59:56.758877 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:56.758841    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:56.759274 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:56.758895    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 14:59:57.758463 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:57.758428    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:57.758666 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:57.758476    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 14:59:58.758420 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:58.758380    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:58.758923 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:58.758437    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 14:59:59.758849 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:59.758813    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 31 14:59:59.759255 ip-10-0-139-115 kubenswrapper[4549]: I1031 14:59:59.758865    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 31 15:00:00.188763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.188729    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:00.193667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.193641    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.245158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.245121    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 15:00:00.285609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.285561    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.285798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.285768    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqdg\" (UniqueName: \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.380912 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.380872    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:00.387027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.386993    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqdg\" (UniqueName: \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.387209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.387123    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.387816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.387796    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.443935 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.443869    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqdg\" (UniqueName: \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.509247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.509213    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:00.763307 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.763211    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 15:00:00.763757 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.763514    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 15:00:00.764340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.764321    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 15:00:00.884905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.884876    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:00.886468 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:00:00.886427    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f53be1_f2e7_487e_be20_6cc04b9edf39.slice/crio-609de990d9fae8e43738b25ba155fb619141d224576709eed5f469a693c1ace6 WatchSource:0}: Error finding container 609de990d9fae8e43738b25ba155fb619141d224576709eed5f469a693c1ace6: Status 404 returned error can't find the container with id 609de990d9fae8e43738b25ba155fb619141d224576709eed5f469a693c1ace6
Oct 31 15:00:00.908548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:00.908510    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerStarted","Data":"609de990d9fae8e43738b25ba155fb619141d224576709eed5f469a693c1ace6"}
Oct 31 15:00:01.917464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:01.917431    4549 generic.go:356] "Generic (PLEG): container finished" podID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerID="ebfe33f819aa56887f6a74762467373c90ab530dd3a604f9c96bea72b5c13484" exitCode=60
Oct 31 15:00:01.917949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:01.917513    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerDied","Data":"ebfe33f819aa56887f6a74762467373c90ab530dd3a604f9c96bea72b5c13484"}
Oct 31 15:00:01.917949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:01.917776    4549 scope.go:122] "RemoveContainer" containerID="ebfe33f819aa56887f6a74762467373c90ab530dd3a604f9c96bea72b5c13484"
Oct 31 15:00:02.866161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.866116    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"]
Oct 31 15:00:02.874134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.874104    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:02.924407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.924375    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"]
Oct 31 15:00:02.928977 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.928948    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 15:00:02.933414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.933384    4549 generic.go:356] "Generic (PLEG): container finished" podID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerID="4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60" exitCode=60
Oct 31 15:00:02.933624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.933540    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerDied","Data":"4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60"}
Oct 31 15:00:02.933713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.933624    4549 scope.go:122] "RemoveContainer" containerID="ebfe33f819aa56887f6a74762467373c90ab530dd3a604f9c96bea72b5c13484"
Oct 31 15:00:02.933907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:02.933884    4549 scope.go:122] "RemoveContainer" containerID="4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60"
Oct 31 15:00:02.934213 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:00:02.934185    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n_odh-ai-gateway-infra(44f53be1-f2e7-487e-be20-6cc04b9edf39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39"
Oct 31 15:00:03.016045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.015659    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.016045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.015717    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.016045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.015827    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgrt\" (UniqueName: \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.117005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.116903    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.117005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.116966    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.117236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.117060    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgrt\" (UniqueName: \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.117591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.117552    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.119796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.119768    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.174133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.174095    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgrt\" (UniqueName: \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\") pod \"maas-api-revoke-sub-a975402d-jtsqc\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.188926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.188895    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:03.578672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.578638    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"]
Oct 31 15:00:03.579404 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:00:03.579381    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4cc62b_c1f8_4752_8bc9_182a8225fcca.slice/crio-cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5 WatchSource:0}: Error finding container cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5: Status 404 returned error can't find the container with id cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5
Oct 31 15:00:03.942914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.942887    4549 scope.go:122] "RemoveContainer" containerID="4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60"
Oct 31 15:00:03.943421 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:00:03.943142    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n_odh-ai-gateway-infra(44f53be1-f2e7-487e-be20-6cc04b9edf39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39"
Oct 31 15:00:03.944496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.944467    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc" event={"ID":"0a4cc62b-c1f8-4752-8bc9-182a8225fcca","Type":"ContainerStarted","Data":"c22f0954ef887c9f0b440ea2c2b9f93eeb15889c6d442e8606d13e5b8082b2be"}
Oct 31 15:00:03.944640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:03.944515    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc" event={"ID":"0a4cc62b-c1f8-4752-8bc9-182a8225fcca","Type":"ContainerStarted","Data":"cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5"}
Oct 31 15:00:04.168709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:04.168603    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc" podStartSLOduration=2.168562625 podStartE2EDuration="2.168562625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:04.167228884 +0000 UTC m=+844.893509865" watchObservedRunningTime="2026-10-31 15:00:04.168562625 +0000 UTC m=+844.894843618"
Oct 31 15:00:04.953028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:04.952997    4549 generic.go:356] "Generic (PLEG): container finished" podID="0a4cc62b-c1f8-4752-8bc9-182a8225fcca" containerID="c22f0954ef887c9f0b440ea2c2b9f93eeb15889c6d442e8606d13e5b8082b2be" exitCode=0
Oct 31 15:00:04.953439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:04.953094    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc" event={"ID":"0a4cc62b-c1f8-4752-8bc9-182a8225fcca","Type":"ContainerDied","Data":"c22f0954ef887c9f0b440ea2c2b9f93eeb15889c6d442e8606d13e5b8082b2be"}
Oct 31 15:00:06.106283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.106253    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:06.249383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.249286    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\") pod \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") "
Oct 31 15:00:06.249383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.249334    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\" (UniqueName: \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\") pod \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") "
Oct 31 15:00:06.249635 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.249420    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\") pod \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\" (UID: \"0a4cc62b-c1f8-4752-8bc9-182a8225fcca\") "
Oct 31 15:00:06.249851 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.249817    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca" pod "0a4cc62b-c1f8-4752-8bc9-182a8225fcca" (UID: "0a4cc62b-c1f8-4752-8bc9-182a8225fcca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:00:06.251830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.251799    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt" pod "0a4cc62b-c1f8-4752-8bc9-182a8225fcca" (UID: "0a4cc62b-c1f8-4752-8bc9-182a8225fcca"). InnerVolumeSpecName "kube-api-access-grgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:00:06.251830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.251822    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca" pod "0a4cc62b-c1f8-4752-8bc9-182a8225fcca" (UID: "0a4cc62b-c1f8-4752-8bc9-182a8225fcca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:00:06.350747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.350709    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-xks-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:00:06.350747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.350739    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgrt\" (UniqueName: \"kubernetes.io/projected/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-kube-api-access-grgrt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:00:06.350747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.350752    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a4cc62b-c1f8-4752-8bc9-182a8225fcca-maas-api-openshift-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:00:06.966615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.966558    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"
Oct 31 15:00:06.966615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.966600    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc" event={"ID":"0a4cc62b-c1f8-4752-8bc9-182a8225fcca","Type":"ContainerDied","Data":"cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5"}
Oct 31 15:00:06.966830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:06.966650    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf613dbaa9177b8c6322e4a614a2678e428172e1c4644a6c76a43b08f67d58e5"
Oct 31 15:00:07.012852 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:00:07.012814    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:00:14.413023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:14.412940    4549 scope.go:122] "RemoveContainer" containerID="4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60"
Oct 31 15:00:15.017504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:15.017469    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerStarted","Data":"1f1a7e1dc26e0fb9520a3bbd2d2bc792b0e2877b7681d2682705243ecea43f93"}
Oct 31 15:00:15.130080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:15.130019    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" podStartSLOduration=15.130000822 podStartE2EDuration="15.130000822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:15.129708825 +0000 UTC m=+855.855989799" watchObservedRunningTime="2026-10-31 15:00:15.130000822 +0000 UTC m=+855.856281801"
Oct 31 15:00:16.025714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:16.025681    4549 generic.go:356] "Generic (PLEG): container finished" podID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerID="1f1a7e1dc26e0fb9520a3bbd2d2bc792b0e2877b7681d2682705243ecea43f93" exitCode=60
Oct 31 15:00:16.026234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:16.025758    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerDied","Data":"1f1a7e1dc26e0fb9520a3bbd2d2bc792b0e2877b7681d2682705243ecea43f93"}
Oct 31 15:00:16.026234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:16.025823    4549 scope.go:122] "RemoveContainer" containerID="4969bdfe6bd6ddb8ccc3239af2c7eabbd1cc8f1c6303bfe1ea199d6f29da3b60"
Oct 31 15:00:16.026234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:16.026198    4549 scope.go:122] "RemoveContainer" containerID="1f1a7e1dc26e0fb9520a3bbd2d2bc792b0e2877b7681d2682705243ecea43f93"
Oct 31 15:00:16.026502 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:00:16.026475    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n_odh-ai-gateway-infra(44f53be1-f2e7-487e-be20-6cc04b9edf39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39"
Oct 31 15:00:16.142931 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:16.142890    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:17.179089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.179067    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:17.266716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.266683    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\") pod \"44f53be1-f2e7-487e-be20-6cc04b9edf39\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") "
Oct 31 15:00:17.266919 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.266732    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\" (UniqueName: \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\") pod \"44f53be1-f2e7-487e-be20-6cc04b9edf39\" (UID: \"44f53be1-f2e7-487e-be20-6cc04b9edf39\") "
Oct 31 15:00:17.267034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.267011    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca" pod "44f53be1-f2e7-487e-be20-6cc04b9edf39" (UID: "44f53be1-f2e7-487e-be20-6cc04b9edf39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:00:17.267117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.267101    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44f53be1-f2e7-487e-be20-6cc04b9edf39-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:00:17.269088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.269063    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg" pod "44f53be1-f2e7-487e-be20-6cc04b9edf39" (UID: "44f53be1-f2e7-487e-be20-6cc04b9edf39"). InnerVolumeSpecName "kube-api-access-7lqdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:00:17.368184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:17.368112    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lqdg\" (UniqueName: \"kubernetes.io/projected/44f53be1-f2e7-487e-be20-6cc04b9edf39-kube-api-access-7lqdg\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:00:18.041676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:18.041644    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"
Oct 31 15:00:18.041834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:18.041677    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n" event={"ID":"44f53be1-f2e7-487e-be20-6cc04b9edf39","Type":"ContainerDied","Data":"609de990d9fae8e43738b25ba155fb619141d224576709eed5f469a693c1ace6"}
Oct 31 15:00:18.041834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:18.041741    4549 scope.go:122] "RemoveContainer" containerID="1f1a7e1dc26e0fb9520a3bbd2d2bc792b0e2877b7681d2682705243ecea43f93"
Oct 31 15:00:18.205623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:18.205593    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:18.209891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:18.209863    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29890980-s785n"]
Oct 31 15:00:19.419148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:19.419116    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" path="/var/lib/kubelet/pods/44f53be1-f2e7-487e-be20-6cc04b9edf39/volumes"
Oct 31 15:00:35.691969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.691936    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 15:00:35.692480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692464    4549 cpu_manager.go:414] "containerMap: removing container" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692483    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692495    4549 cpu_manager.go:414] "containerMap: removing container" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692500    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692526    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0a4cc62b-c1f8-4752-8bc9-182a8225fcca" containerName="revoke-keys"
Oct 31 15:00:35.692533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692534    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4cc62b-c1f8-4752-8bc9-182a8225fcca" containerName="revoke-keys"
Oct 31 15:00:35.692711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692625    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692637    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4cc62b-c1f8-4752-8bc9-182a8225fcca" containerName="revoke-keys"
Oct 31 15:00:35.692711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692651    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.692711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.692657    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:35.702265 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.698298    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.751387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.751338    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\""
Oct 31 15:00:35.762130 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762094    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762171    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd7q\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762220    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762254    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762297    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762343    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762370    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762424    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.762496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.762490    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.805283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.805249    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 15:00:35.864028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.863983    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864053    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864098    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864123    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864177    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864241    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864299    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864355    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd7q\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864397    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864703    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.864783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.864746    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.865026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.865005    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.865312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.865288    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.865777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.865752    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.866741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.866718    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.867259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.867240    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.922084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.922050    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd7q\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:35.922528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:35.922504    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:36.017331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:36.017275    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:36.462863 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:36.462831    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 15:00:36.464951 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:00:36.464923    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc710aa4_1d7a_4421_a876_e5a482292056.slice/crio-ceb42442b71043bffb20da64f04e73b9e77df0c705bd4601a3cf5a596d702fbe WatchSource:0}: Error finding container ceb42442b71043bffb20da64f04e73b9e77df0c705bd4601a3cf5a596d702fbe: Status 404 returned error can't find the container with id ceb42442b71043bffb20da64f04e73b9e77df0c705bd4601a3cf5a596d702fbe
Oct 31 15:00:36.467713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:36.467676    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:36.467785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:36.467748    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:36.467822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:36.467789    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:37.060620 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:00:37.059820    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:00:37.175852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:37.175814    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" event={"ID":"bc710aa4-1d7a-4421-a876-e5a482292056","Type":"ContainerStarted","Data":"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"}
Oct 31 15:00:37.176030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:37.175878    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" event={"ID":"bc710aa4-1d7a-4421-a876-e5a482292056","Type":"ContainerStarted","Data":"ceb42442b71043bffb20da64f04e73b9e77df0c705bd4601a3cf5a596d702fbe"}
Oct 31 15:00:37.336663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:37.336216    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podStartSLOduration=2.336192381 podStartE2EDuration="2.336192381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:37.297268322 +0000 UTC m=+878.023549303" watchObservedRunningTime="2026-10-31 15:00:37.336192381 +0000 UTC m=+878.062473359"
Oct 31 15:00:38.021498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:38.020641    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:38.022705 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:38.022672    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:38.022831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:38.022731    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:39.024476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:39.023947    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:39.024476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:39.024008    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:40.019597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:40.018019    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:40.019597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:40.018085    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:41.017993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:41.017953    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:41.018638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:41.018606    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:42.021597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:42.017968    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:42.021597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:42.018061    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:43.018720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:43.018668    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 31 15:00:43.019057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:43.019010    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 31 15:00:44.025882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:44.025850    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:44.027697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:44.027668    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:44.032803 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:44.032781    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 15:00:55.731182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.731149    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:00:55.731628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.731613    4549 cpu_manager.go:414] "containerMap: removing container" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:55.731628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.731625    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f53be1-f2e7-487e-be20-6cc04b9edf39" containerName="cleanup"
Oct 31 15:00:55.735024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.735009    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.780599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.780557    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:00:55.784316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.784298    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795196 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795173    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795302 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795208    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795302 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795234    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795316    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795399    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795507    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795536    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795631    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.795825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795722    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795779    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795819    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795853    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpkd\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795905    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.795953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795942    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.796089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.795966    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.796089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.796018    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.796159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.796092    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.796159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.796122    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s55b\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.881051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.881020    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:00:55.896913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.896885    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.896924    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897004    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897048    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897123    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897150    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897195    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897240    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897267    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897310    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897367    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpkd\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897408    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897442    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897480    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897512    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897530    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897543    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897612    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s55b\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897688    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.898093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.897948    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.898323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898097    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.898420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898398    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.898701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898676    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.898792 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898761    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.898896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898864    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.898966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.898930    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.899094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.899065    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.899236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.899127    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.899976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.899949    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.900145 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.900129    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.901389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.901368    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.901482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.901422    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.943915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.943869    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:00:55.955263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.955236    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s55b\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.955607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.955563    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:55.955960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.955938    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-jrgbx\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:55.956170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:55.956154    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpkd\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\") pod \"maas-default-gateway-openshift-default-67bf986f99-x27md\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:56.048709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.048667    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:56.096906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.096875    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:56.475323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.475289    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:00:56.476652 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:00:56.476625    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09136301_7149_4162_a5db_8b469f6ba404.slice/crio-6b77c91f60df96e6e455a1bd13a677ff751e2a1c7f1da4e25d0d01473292a147 WatchSource:0}: Error finding container 6b77c91f60df96e6e455a1bd13a677ff751e2a1c7f1da4e25d0d01473292a147: Status 404 returned error can't find the container with id 6b77c91f60df96e6e455a1bd13a677ff751e2a1c7f1da4e25d0d01473292a147
Oct 31 15:00:56.479855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.479813    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:56.479959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.479903    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:56.479959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.479945    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:56.532557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.532529    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:00:56.536505 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:00:56.536473    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821afb3c_91c1_4936_be5b_ff18da551e27.slice/crio-26d06399ab624fc5d0ae95682cd5eac74a7121378f89372f80fe0d9b571f982d WatchSource:0}: Error finding container 26d06399ab624fc5d0ae95682cd5eac74a7121378f89372f80fe0d9b571f982d: Status 404 returned error can't find the container with id 26d06399ab624fc5d0ae95682cd5eac74a7121378f89372f80fe0d9b571f982d
Oct 31 15:00:56.545416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.545377    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:56.545515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.545476    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:56.545597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:56.545525    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:00:57.377284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:57.377247    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" event={"ID":"821afb3c-91c1-4936-be5b-ff18da551e27","Type":"ContainerStarted","Data":"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"}
Oct 31 15:00:57.377777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:57.377306    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" event={"ID":"821afb3c-91c1-4936-be5b-ff18da551e27","Type":"ContainerStarted","Data":"26d06399ab624fc5d0ae95682cd5eac74a7121378f89372f80fe0d9b571f982d"}
Oct 31 15:00:57.379510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:57.379484    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" event={"ID":"09136301-7149-4162-a5db-8b469f6ba404","Type":"ContainerStarted","Data":"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"}
Oct 31 15:00:57.379662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:57.379524    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" event={"ID":"09136301-7149-4162-a5db-8b469f6ba404","Type":"ContainerStarted","Data":"6b77c91f60df96e6e455a1bd13a677ff751e2a1c7f1da4e25d0d01473292a147"}
Oct 31 15:00:57.611645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:57.611585    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podStartSLOduration=2.611552764 podStartE2EDuration="2.611552764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:57.552032968 +0000 UTC m=+898.278313948" watchObservedRunningTime="2026-10-31 15:00:57.611552764 +0000 UTC m=+898.337833743"
Oct 31 15:00:58.048871 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.048817    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:00:58.051390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.051347    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:00:58.051670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.051630    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:00:58.100791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.100734    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:00:58.102900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.102861    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:00:58.103115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.103091    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:00:58.730620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.729744    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podStartSLOduration=3.729721171 podStartE2EDuration="3.729721171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:57.681961026 +0000 UTC m=+898.408242006" watchObservedRunningTime="2026-10-31 15:00:58.729721171 +0000 UTC m=+899.456002151"
Oct 31 15:00:58.733274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.732392    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:00:58.972596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.972283    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:00:58.972596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:58.972417    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.034650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.013709    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.038594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.034905    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.047714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.047670    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.052594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.047958    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.063597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.056698    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.068592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.063904    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznj7\" (UniqueName: \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.072712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.048138    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"e2e-conflict-a-fec0c949-kserve-self-signed-certs\""
Oct 31 15:00:59.076637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.049452    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:00:59.103595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.080654    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:00:59.146591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.143892    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:00:59.146591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.143955    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:00:59.146591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.144430    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"default-dockercfg-hc4sm\""
Oct 31 15:00:59.146591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.144526    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"kube-root-ca.crt\""
Oct 31 15:00:59.146591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.144624    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"openshift-service-ca.crt\""
Oct 31 15:00:59.194598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.172296    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznj7\" (UniqueName: \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.195078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.195046    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.195698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.195663    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.195879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.195862    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.196052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.196037    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.196224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.196207    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.196729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.196710    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.197096 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.197078    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.215731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.200604    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.216693 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.216664    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.246935 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.246198    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.247818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.247762    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznj7\" (UniqueName: \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\") pod \"e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:00:59.497092 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.496997    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"default-dockercfg-hc4sm\""
Oct 31 15:00:59.498194 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:00:59.498171    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:01:00.057618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.057063    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:00.057618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.057150    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:00.102758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.102713    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:00.106664 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.106621    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:00.301687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.301657    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:01:00.331789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:00.331692    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:01:01.009264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.004925    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:01:01.079597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.071488    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:01.079597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.071547    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:01.106598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.100520    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:01.106598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.100603    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:01.868873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.868789    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" event={"ID":"e7201c59-7d82-4788-b28e-ca6e14e6a9b5","Type":"ContainerStarted","Data":"42f3a92e6f57bdb59acaefd995c2fb1915af95adefaf1df51011b2afd175e380"}
Oct 31 15:01:01.868873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.868831    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" event={"ID":"e7201c59-7d82-4788-b28e-ca6e14e6a9b5","Type":"ContainerStarted","Data":"254924fa5b0814aaeab227ce3c8c3fbd50cea824c00cf79d3da56e042a0d7fa8"}
Oct 31 15:01:01.985610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:01.985542    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" podStartSLOduration=3.985524618 podStartE2EDuration="3.985524618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:01.982476224 +0000 UTC m=+902.708757231" watchObservedRunningTime="2026-10-31 15:01:01.985524618 +0000 UTC m=+902.711805621"
Oct 31 15:01:02.050314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:02.049967    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:02.050314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:02.050015    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:02.104006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:02.103967    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:02.104636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:02.104601    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:02.501758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:02.501723    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:01:03.056969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:03.056936    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:03.057221 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:03.057192    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:03.097554 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:03.097519    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:03.097796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:03.097619    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:04.049830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:04.049800    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:04.050251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:04.049859    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:04.097739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:04.097701    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:04.097883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:04.097765    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:05.049790 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:05.049750    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 15:01:05.049965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:05.049822    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 15:01:05.098158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:05.098119    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:05.098310 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:05.098185    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:06.098168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:06.098120    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 15:01:06.098726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:06.098193    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 15:01:06.989248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:06.988897    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:01:06.990328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:06.990264    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:01:06.991825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:06.991798    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:01:07.024161 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:01:07.024124    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:01:07.461816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:07.461772    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:01:07.462517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:07.462485    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:01:07.463159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:07.463140    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:01:09.501068 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:09.501028    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:01:09.514304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:09.514276    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:01:09.970317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:09.970284    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:01:28.448033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.447992    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:01:28.452945 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.452919    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.476790 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.476754    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477062    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477298    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477450    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477667    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477809    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbt7\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.477966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.477949    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.478133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.478115    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.478255 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.478239    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.506893 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.506803    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2348-openshift-default-dockercfg-k9qcg\""
Oct 31 15:01:28.572916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.572886    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.578968    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579039    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579079    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579124    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579148    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbt7\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579191    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579260    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579289    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579328    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579431    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.579753    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.580460    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.580839    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.581377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.581318    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.582731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.582703    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.585292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.585226    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.638155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.638112    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.639079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.639050    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbt7\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:28.773331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:28.773287    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:29.512594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:29.510902    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:01:29.645316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:29.643782    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:01:29.645316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:29.643885    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:01:29.645316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:29.643932    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:01:30.280486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.280447    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" event={"ID":"ac19db83-4bfe-4282-a380-e5143b46e48c","Type":"ContainerStarted","Data":"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"}
Oct 31 15:01:30.280821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.280798    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" event={"ID":"ac19db83-4bfe-4282-a380-e5143b46e48c","Type":"ContainerStarted","Data":"f9083c1c6758218a0c356ff4bd0e8c6b73e077b989701e1c9a9ec84f4dfcdce9"}
Oct 31 15:01:30.720767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.720661    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:01:30.777775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.777635    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:30.781719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.781682    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:30.781994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:30.781965    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:31.774087 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:31.774040    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:31.774808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:31.774776    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:32.773739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:32.773702    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:32.773900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:32.773772    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:33.774281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:33.774247    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:33.774733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:33.774299    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:34.774172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:34.774129    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:34.774384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:34.774184    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:35.774064 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:35.774025    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:35.774245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:35.774081    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:36.774357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:36.774317    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:36.774787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:36.774379    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:37.027067 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:01:37.026975    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:01:37.773962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:37.773929    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 31 15:01:37.774242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:37.773988    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 31 15:01:38.778375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:38.778337    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:38.778989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:38.778969    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:38.780232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:38.780211    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:01:38.946172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:38.946115    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podStartSLOduration=10.946100749 podStartE2EDuration="10.946100749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:30.514871107 +0000 UTC m=+931.241152087" watchObservedRunningTime="2026-10-31 15:01:38.946100749 +0000 UTC m=+939.672381727"
Oct 31 15:01:46.521121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.521036    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 15:01:46.524785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.524764    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.597127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.597096    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"openshift-service-ca.crt\""
Oct 31 15:01:46.627112 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.627082    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"kube-root-ca.crt\""
Oct 31 15:01:46.627324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.627219    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"default-dockercfg-vznpg\""
Oct 31 15:01:46.627324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.627259    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"distinct-w2-029d26-kserve-self-signed-certs\""
Oct 31 15:01:46.632135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.632114    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 15:01:46.674013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.673981    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.674013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.674019    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.674285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.674179    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x689l\" (UniqueName: \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.674285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.674236    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.674405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.674283    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.674405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.674308    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.775715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775609    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.775715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775650    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.775715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775712    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x689l\" (UniqueName: \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775740    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775768    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.775791    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776202 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.776164    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.776203    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.776416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.776391    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.778316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.778292    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.778509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.778495    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.832505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.832473    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x689l\" (UniqueName: \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\") pod \"distinct-w2-029d26-kserve-54c5df7dd8-qbkbn\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:46.839248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:46.839166    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:47.240499 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:47.240455    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 15:01:47.242040 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:01:47.242000    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1240204_96ae_4de2_a443_af97313eec1b.slice/crio-db28e1935ce841737783fedc2769190ccb61ede85e5343001e0afbd3366f9fb7 WatchSource:0}: Error finding container db28e1935ce841737783fedc2769190ccb61ede85e5343001e0afbd3366f9fb7: Status 404 returned error can't find the container with id db28e1935ce841737783fedc2769190ccb61ede85e5343001e0afbd3366f9fb7
Oct 31 15:01:47.526636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:47.526600    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" event={"ID":"a1240204-96ae-4de2-a443-af97313eec1b","Type":"ContainerStarted","Data":"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"}
Oct 31 15:01:47.527151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:47.526646    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" event={"ID":"a1240204-96ae-4de2-a443-af97313eec1b","Type":"ContainerStarted","Data":"db28e1935ce841737783fedc2769190ccb61ede85e5343001e0afbd3366f9fb7"}
Oct 31 15:01:47.644660 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:47.644592    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podStartSLOduration=1.644553454 podStartE2EDuration="1.644553454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:47.64046198 +0000 UTC m=+948.366742961" watchObservedRunningTime="2026-10-31 15:01:47.644553454 +0000 UTC m=+948.370834434"
Oct 31 15:01:47.839756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:47.839659    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:54.216747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.216710    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:01:54.220782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.220760    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.322769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.322715    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-57815b\"/\"distinct-w4-57815b-kserve-self-signed-certs\""
Oct 31 15:01:54.327018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.326991    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:01:54.363901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.363861    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.364125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.363968    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.364125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.364037    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.364125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.364086    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bg94\" (UniqueName: \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.364125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.364116    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.364353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.364176    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.464953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.464918    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.464975    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465013    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bg94\" (UniqueName: \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465037    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465084    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465164    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465426    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465488    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.465644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.465621    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.467503 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.467449    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.467907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.467877    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.522480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.522445    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bg94\" (UniqueName: \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\") pod \"distinct-w4-57815b-kserve-57c65f547c-pgrqs\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") " pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.533294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.533260    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:54.942208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:54.942173    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:01:54.944144 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:01:54.944119    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b2c1027_29aa_4c9e_95f1_4fea89d70633.slice/crio-00b55f3fa8bc25b2c3d801e51ddeade8c2415bde1e6bd2a91ecb5f09ba78eab3 WatchSource:0}: Error finding container 00b55f3fa8bc25b2c3d801e51ddeade8c2415bde1e6bd2a91ecb5f09ba78eab3: Status 404 returned error can't find the container with id 00b55f3fa8bc25b2c3d801e51ddeade8c2415bde1e6bd2a91ecb5f09ba78eab3
Oct 31 15:01:55.578640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:55.578602    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" event={"ID":"8b2c1027-29aa-4c9e-95f1-4fea89d70633","Type":"ContainerStarted","Data":"69d1534ff5cb73420271f9284018ce15e08029e6e3d8babdd71888803252017f"}
Oct 31 15:01:55.578640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:55.578647    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" event={"ID":"8b2c1027-29aa-4c9e-95f1-4fea89d70633","Type":"ContainerStarted","Data":"00b55f3fa8bc25b2c3d801e51ddeade8c2415bde1e6bd2a91ecb5f09ba78eab3"}
Oct 31 15:01:55.693209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:55.693160    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podStartSLOduration=1.693146455 podStartE2EDuration="1.693146455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:55.692564234 +0000 UTC m=+956.418845215" watchObservedRunningTime="2026-10-31 15:01:55.693146455 +0000 UTC m=+956.419427432"
Oct 31 15:01:56.533670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:56.533642    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:01:56.839952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:56.839855    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:56.853400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:56.853375    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:01:57.602900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:01:57.602873    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:02:04.533760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:04.533727    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:02:04.546099 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:04.546073    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:02:04.653546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:04.653514    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:02:07.014562 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:02:07.014525    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:02:19.961797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:19.961761    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:02:19.966080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:19.966060    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062347    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062398    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062432    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5c8l\" (UniqueName: \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062563    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062642    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.062734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.062664    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.068712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.068690    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"e2e-conflict-b-fec0c949-kserve-self-signed-certs\""
Oct 31 15:02:20.071750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.071729    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:02:20.164280 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164245    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164305    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164342    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5c8l\" (UniqueName: \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164391    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164439    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164466    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164807    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.164962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164853    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.165012 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.164968    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.166887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.166867    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.167010 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.166993    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.221583 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.221484    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5c8l\" (UniqueName: \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\") pod \"e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") " pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.279958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.279922    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:20.666048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.666025    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:02:20.666893 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:02:20.666864    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc52ad659_ef0d_4614_8f67_8790034943c7.slice/crio-ab327ed7973577fb475077ca0325677074dbc8e638c0eeed464d47b08bb7ab5a WatchSource:0}: Error finding container ab327ed7973577fb475077ca0325677074dbc8e638c0eeed464d47b08bb7ab5a: Status 404 returned error can't find the container with id ab327ed7973577fb475077ca0325677074dbc8e638c0eeed464d47b08bb7ab5a
Oct 31 15:02:20.744637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.744604    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" event={"ID":"c52ad659-ef0d-4614-8f67-8790034943c7","Type":"ContainerStarted","Data":"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"}
Oct 31 15:02:20.744771 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.744646    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" event={"ID":"c52ad659-ef0d-4614-8f67-8790034943c7","Type":"ContainerStarted","Data":"ab327ed7973577fb475077ca0325677074dbc8e638c0eeed464d47b08bb7ab5a"}
Oct 31 15:02:20.858556 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:20.858498    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" podStartSLOduration=1.858484554 podStartE2EDuration="1.858484554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:02:20.856625919 +0000 UTC m=+981.582906897" watchObservedRunningTime="2026-10-31 15:02:20.858484554 +0000 UTC m=+981.584765540"
Oct 31 15:02:21.280107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:21.280066    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:23.608758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.608727    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:02:23.612625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.612595    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.711747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711699    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.711944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711780    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.711944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711853    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.711944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711878    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.711944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711914    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.712200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.711955    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.712200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.712013    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8mdd\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.712200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.712097    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.712200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.712137    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.719409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.719376    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:02:23.813842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813801    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.813842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813836    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813857    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813880    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813923    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8mdd\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.813981    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814017    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814075    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814121    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814255    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814488    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814624    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.814724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.814683    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.815201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.815179    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.816513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.816494    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.816754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.816731    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.870921 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.870839    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.871086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.871048    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8mdd\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:23.926706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:23.926664    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:24.373915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.373865    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:02:24.399767 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:02:24.399736    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3154a4_64c0_497f_b7a6_005ace06c78c.slice/crio-f7ec077046268538fbca437854985502a277d65e2d46e61d948fe1bf6bc174fd WatchSource:0}: Error finding container f7ec077046268538fbca437854985502a277d65e2d46e61d948fe1bf6bc174fd: Status 404 returned error can't find the container with id f7ec077046268538fbca437854985502a277d65e2d46e61d948fe1bf6bc174fd
Oct 31 15:02:24.402363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.402318    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:02:24.402452 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.402419    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:02:24.402490 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.402463    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:02:24.780582 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.780546    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" event={"ID":"3b3154a4-64c0-497f-b7a6-005ace06c78c","Type":"ContainerStarted","Data":"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"}
Oct 31 15:02:24.781090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.780614    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" event={"ID":"3b3154a4-64c0-497f-b7a6-005ace06c78c","Type":"ContainerStarted","Data":"f7ec077046268538fbca437854985502a277d65e2d46e61d948fe1bf6bc174fd"}
Oct 31 15:02:24.900883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.900802    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podStartSLOduration=1.900779905 podStartE2EDuration="1.900779905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:02:24.896243375 +0000 UTC m=+985.622524353" watchObservedRunningTime="2026-10-31 15:02:24.900779905 +0000 UTC m=+985.627060898"
Oct 31 15:02:24.928595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:24.928211    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:25.930304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:25.930262    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:25.930862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:25.930333    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:26.929267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:26.929229    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:26.929488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:26.929297    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:27.928029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:27.927975    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:27.928517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:27.928045    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:28.927924 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:28.927874    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:28.928118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:28.927946    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:29.928038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:29.927992    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:29.928230 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:29.928065    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:30.281067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:30.281033    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:30.294362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:30.294338    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:30.870244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:30.870213    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:02:30.927319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:30.927285    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 15:02:30.927485 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:30.927348    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 15:02:31.933081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:31.933053    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:31.934031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:31.934006    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:31.935032 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:31.935005    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:02:37.012417 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:02:37.012382    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:02:50.497715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.497652    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 15:02:50.501905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.501882    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.604098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.604067    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-029d26\"/\"distinct-2-w2-029d26-kserve-self-signed-certs\""
Oct 31 15:02:50.607096 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.607074    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 15:02:50.613549 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613517    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.613701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613618    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.613701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613654    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.613823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613786    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.613877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613836    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.613950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.613930    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqfr\" (UniqueName: \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715426 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715387    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715471    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715503    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715590    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715626    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.715680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.715679    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqfr\" (UniqueName: \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.716277 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.716252    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.716375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.716280    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.716671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.716650    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.719553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.719530    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.723863 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.723837    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.772547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.772510    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqfr\" (UniqueName: \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\") pod \"distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") " pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:50.815089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:50.815052    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:51.245621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:51.245590    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 15:02:51.245898 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:02:51.245876    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbec4cd1e_46c9_45d0_aa8e_49ab69ab6a70.slice/crio-fc38287829180cdaeb6d1eb07e702721ec2395c9725b9321eeb111969e2f5f79 WatchSource:0}: Error finding container fc38287829180cdaeb6d1eb07e702721ec2395c9725b9321eeb111969e2f5f79: Status 404 returned error can't find the container with id fc38287829180cdaeb6d1eb07e702721ec2395c9725b9321eeb111969e2f5f79
Oct 31 15:02:52.008951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:52.008918    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" event={"ID":"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70","Type":"ContainerStarted","Data":"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"}
Oct 31 15:02:52.009469 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:52.008963    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" event={"ID":"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70","Type":"ContainerStarted","Data":"fc38287829180cdaeb6d1eb07e702721ec2395c9725b9321eeb111969e2f5f79"}
Oct 31 15:02:52.126125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:52.126068    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podStartSLOduration=2.126053829 podStartE2EDuration="2.126053829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:02:52.123107157 +0000 UTC m=+1012.849388136" watchObservedRunningTime="2026-10-31 15:02:52.126053829 +0000 UTC m=+1012.852334807"
Oct 31 15:02:52.815778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:52.815741    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:02:59.142429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.142389    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:02:59.146848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.146827    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322150    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322213    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322379    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322490 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322436    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322599    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322689 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322643    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322777    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322826    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.322963 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.322945    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96m8\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.349663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.349626    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:02:59.423743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423651    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.423743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423695    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.423743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423734    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423755    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423823    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423853    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423912    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96m8\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.423974    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.424004    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.424068    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.424589    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.424768    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.424970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.424945    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.425247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.425226    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.426190 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.426171    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.426406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.426387    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.481350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.481315    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.481486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.481427    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96m8\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.762396 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.762310    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:02:59.881766 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.881738    4549 scope.go:122] "RemoveContainer" containerID="91a0d55784c9b9b95e504635aab249aa0c72914d3ed4fa6372066825e16e64ed"
Oct 31 15:02:59.894292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:02:59.894273    4549 scope.go:122] "RemoveContainer" containerID="18d534ecc2fc3a422b2a8c1063ac97e86054e56d34122c47f4b19845b573741b"
Oct 31 15:03:00.193682 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.193654    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:03:00.194508 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:03:00.194482    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2812e9f_67cc_4190_87e7_daed6ac347a4.slice/crio-32909758ba469cb6ce560ad3a5df82dbca82b5ed46973df1fe282d4ca172066b WatchSource:0}: Error finding container 32909758ba469cb6ce560ad3a5df82dbca82b5ed46973df1fe282d4ca172066b: Status 404 returned error can't find the container with id 32909758ba469cb6ce560ad3a5df82dbca82b5ed46973df1fe282d4ca172066b
Oct 31 15:03:00.196978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.196944    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:00.197084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.197038    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:00.197084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.197075    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:00.820594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.817982    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:03:00.836432 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:00.836395    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:03:01.073749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.073658    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" event={"ID":"d2812e9f-67cc-4190-87e7-daed6ac347a4","Type":"ContainerStarted","Data":"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"}
Oct 31 15:03:01.073749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.073714    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" event={"ID":"d2812e9f-67cc-4190-87e7-daed6ac347a4","Type":"ContainerStarted","Data":"32909758ba469cb6ce560ad3a5df82dbca82b5ed46973df1fe282d4ca172066b"}
Oct 31 15:03:01.085650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.085627    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:03:01.192386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.192336    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podStartSLOduration=2.192320837 podStartE2EDuration="2.192320837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:01.189907011 +0000 UTC m=+1021.916187989" watchObservedRunningTime="2026-10-31 15:03:01.192320837 +0000 UTC m=+1021.918601815"
Oct 31 15:03:01.763714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.763672    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:03:01.765133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.765103    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:01.765378 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:01.765348    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:02.763282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:02.763244    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:02.763473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:02.763314    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:03.763663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:03.763622    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:03.764064 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:03.763683    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:04.763507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:04.763476    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:04.763717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:04.763526    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:05.762693 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:05.762655    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:05.762874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:05.762708    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:06.763631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:06.763561    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 15:03:06.764078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:06.763680    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 15:03:07.014100 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:03:07.014005    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:03:08.105942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:08.105896    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:03:08.106874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:08.106852    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:03:08.107848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:08.107827    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:03:27.560852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:27.560770    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:03:34.558940 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.558906    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:03:34.567767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.567742    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668138    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668179    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668287    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrfs\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668326    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668347    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668488    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668601    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668768    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.668941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.668815    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770231    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770285    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770314    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrfs\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770339    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770363    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770424    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770482    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770638    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.770915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.770680    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.771172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.771149    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.771376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.771347    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.771461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.771445    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.772180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.772119    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.772180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.772192    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.773095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.773067    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:03:34.774700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.774679    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.775113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.775092    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.828960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.828890    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.829973 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.829952    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrfs\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:34.883457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:34.883430    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:35.321950 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:03:35.321921    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ab3497_76d8_4b28_b9fb_8bd80ec5f4bf.slice/crio-4d7df347a9f04a88f859e8288e57bd1f1eb342536f4996e88e83d48512b1fccf WatchSource:0}: Error finding container 4d7df347a9f04a88f859e8288e57bd1f1eb342536f4996e88e83d48512b1fccf: Status 404 returned error can't find the container with id 4d7df347a9f04a88f859e8288e57bd1f1eb342536f4996e88e83d48512b1fccf
Oct 31 15:03:35.322238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:35.322213    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:03:35.324463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:35.324427    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:35.324554 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:35.324510    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:35.324623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:35.324555    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:36.335933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:36.335891    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" event={"ID":"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf","Type":"ContainerStarted","Data":"e8f443ba3be8cd1f54c48cb2d15cfb020facfe5fb3cc12703553ee43055ae361"}
Oct 31 15:03:36.336347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:36.335955    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" event={"ID":"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf","Type":"ContainerStarted","Data":"4d7df347a9f04a88f859e8288e57bd1f1eb342536f4996e88e83d48512b1fccf"}
Oct 31 15:03:36.884421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:36.884389    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:36.886060 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:36.886036    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:36.886186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:36.886080    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:37.036512 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:03:37.036474    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:03:37.884248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:37.884207    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:37.884755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:37.884272    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:38.885089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:38.885052    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:38.885753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:38.885696    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:39.885593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:39.884940    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:39.885593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:39.885005    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:40.888600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:40.884899    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:40.888600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:40.884961    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:41.884156 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:41.884125    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:41.884375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:41.884185    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:42.582274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.582235    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" containerName="main" containerID="cri-o://67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26" gracePeriod=45
Oct 31 15:03:42.866287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.866258    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:03:42.884188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.884151    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:03:42.884338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.884197    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:03:42.980653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.980482    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podStartSLOduration=8.980464086 podStartE2EDuration="8.980464086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:36.52196768 +0000 UTC m=+1057.248248662" watchObservedRunningTime="2026-10-31 15:03:42.980464086 +0000 UTC m=+1063.706745066"
Oct 31 15:03:42.981665 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.981632    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.981828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.981708    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\" (UniqueName: \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.981828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.981800    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.981942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.981848    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.981942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.981895    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.982051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.982030    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\") pod \"c52ad659-ef0d-4614-8f67-8790034943c7\" (UID: \"c52ad659-ef0d-4614-8f67-8790034943c7\") "
Oct 31 15:03:42.982779 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.982755    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:03:42.983392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.983349    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:03:42.983496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.983365    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:03:42.985113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.985087    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:03:42.985853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.985827    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "kube-api-access-l5c8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:03:42.986040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:42.986021    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs" pod "c52ad659-ef0d-4614-8f67-8790034943c7" (UID: "c52ad659-ef0d-4614-8f67-8790034943c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:03:43.084167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084131    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.084167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084162    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.084358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084176    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5c8l\" (UniqueName: \"kubernetes.io/projected/c52ad659-ef0d-4614-8f67-8790034943c7-kube-api-access-l5c8l\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.084358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084193    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52ad659-ef0d-4614-8f67-8790034943c7-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.084358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084208    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.084358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.084223    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52ad659-ef0d-4614-8f67-8790034943c7-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:03:43.442118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.442080    4549 generic.go:356] "Generic (PLEG): container finished" podID="c52ad659-ef0d-4614-8f67-8790034943c7" containerID="67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26" exitCode=0
Oct 31 15:03:43.442700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.442680    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"
Oct 31 15:03:43.446528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.446493    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" event={"ID":"c52ad659-ef0d-4614-8f67-8790034943c7","Type":"ContainerDied","Data":"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"}
Oct 31 15:03:43.446697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.446678    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" event={"ID":"c52ad659-ef0d-4614-8f67-8790034943c7","Type":"ContainerDied","Data":"ab327ed7973577fb475077ca0325677074dbc8e638c0eeed464d47b08bb7ab5a"}
Oct 31 15:03:43.446820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.446807    4549 scope.go:122] "RemoveContainer" containerID="67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"
Oct 31 15:03:43.463368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.463342    4549 scope.go:122] "RemoveContainer" containerID="67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"
Oct 31 15:03:43.463700 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:03:43.463680    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26\": container with ID starting with 67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26 not found: ID does not exist" containerID="67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"
Oct 31 15:03:43.463767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.463709    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26"} err="failed to get container status \"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26\": rpc error: code = NotFound desc = could not find container \"67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26\": container with ID starting with 67e1f43a595fb5bf809a5b5ff7cb5a2ab52c2249205fb0e23baafe4f80dbcc26 not found: ID does not exist"
Oct 31 15:03:43.611726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.611689    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:03:43.616898 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.616866    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574"]
Oct 31 15:03:43.859974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.859933    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-b-fec0c949-kserve-558db58fcd-fh574" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:03:43.948832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.948786    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:43.949132 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.949093    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:43.950168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:43.950147    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:03:45.419623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:45.419592    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" path="/var/lib/kubelet/pods/c52ad659-ef0d-4614-8f67-8790034943c7/volumes"
Oct 31 15:03:49.474123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.474086    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:03:49.689387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.689350    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:03:49.690269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.690244    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" containerName="main"
Oct 31 15:03:49.690455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.690429    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" containerName="main"
Oct 31 15:03:49.690636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.690620    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ad659-ef0d-4614-8f67-8790034943c7" containerName="main"
Oct 31 15:03:49.694585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.694546    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.752515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.752428    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:03:49.763797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.763763    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.763972 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.763805    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.763962    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764000    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764121    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764185    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764247    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764274    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.764393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.764377    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8pg\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865049 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865012    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865049 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865050    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865087    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865120    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865154    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865171    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865261    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8pg\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865583 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865309    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865583 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865328    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865691 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865673    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865714    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865829 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865806    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.865914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.865866    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.866155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.866137    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.867908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.867888    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.868001 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.867956    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.921764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.921733    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:49.921916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:49.921840    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8pg\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:50.011206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:50.011109    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:50.445437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:50.445405    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:03:50.483089 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:03:50.483056    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c75f5a_8e37_4bc9_838c_8e081b810a4c.slice/crio-f5e133b096998867e4eb03cfd80c120986a5c2467a6193fa6dbd259ae7fea599 WatchSource:0}: Error finding container f5e133b096998867e4eb03cfd80c120986a5c2467a6193fa6dbd259ae7fea599: Status 404 returned error can't find the container with id f5e133b096998867e4eb03cfd80c120986a5c2467a6193fa6dbd259ae7fea599
Oct 31 15:03:50.485351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:50.485319    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:50.485431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:50.485386    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:50.485431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:50.485416    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:51.495407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:51.495365    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" event={"ID":"d7c75f5a-8e37-4bc9-838c-8e081b810a4c","Type":"ContainerStarted","Data":"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"}
Oct 31 15:03:51.495905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:51.495421    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" event={"ID":"d7c75f5a-8e37-4bc9-838c-8e081b810a4c","Type":"ContainerStarted","Data":"f5e133b096998867e4eb03cfd80c120986a5c2467a6193fa6dbd259ae7fea599"}
Oct 31 15:03:51.661561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:51.661501    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podStartSLOduration=2.661482518 podStartE2EDuration="2.661482518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:51.661081234 +0000 UTC m=+1072.387362213" watchObservedRunningTime="2026-10-31 15:03:51.661482518 +0000 UTC m=+1072.387763497"
Oct 31 15:03:52.012048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:52.012017    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:52.024818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:52.024778    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:52.025147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:52.025114    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:53.018596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:53.017868    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:53.018596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:53.017974    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:54.019998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.019961    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:54.020680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.020640    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:54.326775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.326687    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:03:54.330497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.330478    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439019    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439081    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439228    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439257    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76b5f\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439286    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439311    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439470    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439556    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.439614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.439607    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.468819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.468789    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:03:54.540439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540405    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540461    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540484    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540534    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540704    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540744    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76b5f\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540775    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540818    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540897 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540883    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.540956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540909    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.541007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.540964    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.541245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.541189    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.541586 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.541546    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.541715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.541681    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.543557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.543534    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.543783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.543766    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.597565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.597500    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.597565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.597540    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76b5f\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:54.645465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:54.645430    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:55.017656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.014262    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:55.017656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.014322    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:55.128454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.128322    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:03:55.132744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.132527    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:55.132744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.132651    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:55.132744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.132698    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:03:55.563438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.562703    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" event={"ID":"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2","Type":"ContainerStarted","Data":"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"}
Oct 31 15:03:55.563438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.562817    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" event={"ID":"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2","Type":"ContainerStarted","Data":"1628bd5f9c345c1eca73b406862cef35807630884c7f3178d83b7b9750ae34b3"}
Oct 31 15:03:55.646550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.646517    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:03:55.682354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:55.682295    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podStartSLOduration=1.682282697 podStartE2EDuration="1.682282697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:55.679634292 +0000 UTC m=+1076.405915272" watchObservedRunningTime="2026-10-31 15:03:55.682282697 +0000 UTC m=+1076.408563674"
Oct 31 15:03:56.015651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:56.013974    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:56.015651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:56.014037    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:56.648903 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:56.648872    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:03:56.649372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:56.648930    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:03:57.013052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:57.012956    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 15:03:57.013318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:57.013288    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 15:03:57.646855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:57.646810    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:03:57.647085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:57.646873    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:03:58.657418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:58.657381    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:03:58.658119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:58.658084    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:03:58.697407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:58.697378    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:58.706597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:58.697831    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:58.706597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:58.705112    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:03:59.633597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.633169    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:03:59.640142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.640110    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.645888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.645858    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:03:59.646150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.646113    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:03:59.671835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.671790    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:03:59.736587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.736535    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.736890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.736859    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.736968    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737012    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6d6\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737071    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737175    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737223    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737784    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.737952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.737930    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.839199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.839157    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.839429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.839354    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.839683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.839641    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.839797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.839768    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6d6\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840000 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.839862    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840018    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840023    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840120    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840019    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840172    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840275    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840309    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840656    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.840736 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.840719    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.843087 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.842955    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.843087 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.843001    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.896539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.896462    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.896539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.896487    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6d6\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:03:59.956063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:03:59.956019    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:04:00.435607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.434972    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:04:00.460228 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:04:00.460197    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04cb87e_0164_446c_824c_6740b2f8175f.slice/crio-e2dc55e565861ce03d23cf5563fcb9406443ac1e6806fd134860ad272a3adb2e WatchSource:0}: Error finding container e2dc55e565861ce03d23cf5563fcb9406443ac1e6806fd134860ad272a3adb2e: Status 404 returned error can't find the container with id e2dc55e565861ce03d23cf5563fcb9406443ac1e6806fd134860ad272a3adb2e
Oct 31 15:04:00.462955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.462917    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:04:00.463050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.462992    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:04:00.463050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.463020    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:04:00.649591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.645770    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:04:00.649591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.645832    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:04:00.649591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.649372    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" event={"ID":"c04cb87e-0164-446c-824c-6740b2f8175f","Type":"ContainerStarted","Data":"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"}
Oct 31 15:04:00.649591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.649418    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" event={"ID":"c04cb87e-0164-446c-824c-6740b2f8175f","Type":"ContainerStarted","Data":"e2dc55e565861ce03d23cf5563fcb9406443ac1e6806fd134860ad272a3adb2e"}
Oct 31 15:04:00.772072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.772005    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podStartSLOduration=1.771989636 podStartE2EDuration="1.771989636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:00.770871566 +0000 UTC m=+1081.497152546" watchObservedRunningTime="2026-10-31 15:04:00.771989636 +0000 UTC m=+1081.498270616"
Oct 31 15:04:00.956838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:00.956800    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:04:01.646303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:01.645856    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 15:04:01.646303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:01.645915    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 15:04:01.960462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:01.960373    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:01.960945 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:01.960473    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:02.957721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:02.957680    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:02.958018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:02.957988    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:03.341243 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:03.341197    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:04:03.341830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:03.341809    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:04:03.342701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:03.342681    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:04:03.958443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:03.957972    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:03.958443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:03.958039    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:04.505882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:04.505818    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main" containerID="cri-o://42f3a92e6f57bdb59acaefd995c2fb1915af95adefaf1df51011b2afd175e380" gracePeriod=45
Oct 31 15:04:04.958001 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:04.957960    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:04.958283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:04.958252    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:04.959698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:04.959656    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 31 15:04:05.561629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.556741    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:05.643693 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.640921    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:05.967086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.966039    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:05.967086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.966099    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:05.990593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.988048    4549 generic.go:356] "Generic (PLEG): container finished" podID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerID="42f3a92e6f57bdb59acaefd995c2fb1915af95adefaf1df51011b2afd175e380" exitCode=0
Oct 31 15:04:05.990593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:05.988286    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" event={"ID":"e7201c59-7d82-4788-b28e-ca6e14e6a9b5","Type":"ContainerDied","Data":"42f3a92e6f57bdb59acaefd995c2fb1915af95adefaf1df51011b2afd175e380"}
Oct 31 15:04:06.057152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.055823    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121062    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121138    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121205    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\" (UniqueName: \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121317    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121367    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.121498    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\") pod \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\" (UID: \"e7201c59-7d82-4788-b28e-ca6e14e6a9b5\") "
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.144687    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:04:06.149041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.148660    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:04:06.164596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.156999    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:04:06.164596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.157510    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.164596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.157530    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.164596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.157544    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.253172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.253104    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:04:06.265629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.263526    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.355758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.355710    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:04:06.374199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.374165    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.405295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.404081    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:06.482770 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.482726    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7" pod "e7201c59-7d82-4788-b28e-ca6e14e6a9b5" (UID: "e7201c59-7d82-4788-b28e-ca6e14e6a9b5"). InnerVolumeSpecName "kube-api-access-qznj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:04:06.488795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.488761    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznj7\" (UniqueName: \"kubernetes.io/projected/e7201c59-7d82-4788-b28e-ca6e14e6a9b5-kube-api-access-qznj7\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:04:06.962990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.962952    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:06.963561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.963528    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:06.972702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.972660    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:06.972959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:06.972944    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:04:07.122886 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.122798    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh" event={"ID":"e7201c59-7d82-4788-b28e-ca6e14e6a9b5","Type":"ContainerDied","Data":"254924fa5b0814aaeab227ce3c8c3fbd50cea824c00cf79d3da56e042a0d7fa8"}
Oct 31 15:04:07.123161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.123115    4549 scope.go:122] "RemoveContainer" containerID="42f3a92e6f57bdb59acaefd995c2fb1915af95adefaf1df51011b2afd175e380"
Oct 31 15:04:07.124761 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.124741    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"
Oct 31 15:04:07.194217 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:04:07.194184    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:04:07.340058 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.340027    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:04:07.390081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.389996    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/e2e-conflict-a-fec0c949-kserve-5f65b48-xhwvh"]
Oct 31 15:04:07.420528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.420498    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" path="/var/lib/kubelet/pods/e7201c59-7d82-4788-b28e-ca6e14e6a9b5/volumes"
Oct 31 15:04:07.957386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.957349    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 15:04:07.957599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:07.957410    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 15:04:09.798373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:09.798346    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:04:09.798784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:09.798613    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:04:09.799585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:09.799553    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:04:23.769798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:23.769750    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:24.383764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:24.383712    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:37.015954 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:04:37.015914    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:04:40.535091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.535007    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 15:04:40.535785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.535765    4549 cpu_manager.go:414] "containerMap: removing container" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main"
Oct 31 15:04:40.535907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.535786    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main"
Oct 31 15:04:40.535975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.535906    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7201c59-7d82-4788-b28e-ca6e14e6a9b5" containerName="main"
Oct 31 15:04:40.540560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.540538    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.598054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.598021    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 15:04:40.640909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.640884    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"openshift-service-ca.crt\""
Oct 31 15:04:40.641091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.640933    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"default-dockercfg-7gwx9\""
Oct 31 15:04:40.641091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.640953    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"kube-root-ca.crt\""
Oct 31 15:04:40.641830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.641809    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"distinct-2-w3-190693-kserve-self-signed-certs\""
Oct 31 15:04:40.728644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.728564    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.728855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.728780    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.728855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.728809    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.729031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.728872    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qwx\" (UniqueName: \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.729031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.728977    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.729274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.729215    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.830964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.830874    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.830964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.830934    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.830964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.830962    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.830985    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qwx\" (UniqueName: \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.831026    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.831250    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.831342    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.831410    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.831548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.831526    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.833422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.833401    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.833737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.833719    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:40.900589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:40.899546    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qwx\" (UniqueName: \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\") pod \"distinct-2-w3-190693-kserve-848c56b577-fgswh\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") " pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:41.155137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:41.155052    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:41.533136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:41.533107    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 15:04:41.534540 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:04:41.534513    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de67f02_20c2_4163_b8b2_961280068c46.slice/crio-9c6f798c321605eebcca651b6420edcecef3d93afa370604369e52c218c4732a WatchSource:0}: Error finding container 9c6f798c321605eebcca651b6420edcecef3d93afa370604369e52c218c4732a: Status 404 returned error can't find the container with id 9c6f798c321605eebcca651b6420edcecef3d93afa370604369e52c218c4732a
Oct 31 15:04:41.561312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:41.561284    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" event={"ID":"2de67f02-20c2-4163-b8b2-961280068c46","Type":"ContainerStarted","Data":"9c6f798c321605eebcca651b6420edcecef3d93afa370604369e52c218c4732a"}
Oct 31 15:04:42.570411 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:42.570363    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" event={"ID":"2de67f02-20c2-4163-b8b2-961280068c46","Type":"ContainerStarted","Data":"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"}
Oct 31 15:04:42.687783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:42.687714    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podStartSLOduration=2.687693926 podStartE2EDuration="2.687693926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:42.683410969 +0000 UTC m=+1123.409691947" watchObservedRunningTime="2026-10-31 15:04:42.687693926 +0000 UTC m=+1123.413974906"
Oct 31 15:04:43.155786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:43.155750    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:44.220712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.220675    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 15:04:44.225450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.225427    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277519 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277489    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277519 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277522    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l847w\" (UniqueName: \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277656    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277717    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277886 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277833    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.277937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.277888    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.279353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.279328    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 15:04:44.325147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.325103    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"distinct-w1-fb2348-kserve-self-signed-certs\""
Oct 31 15:04:44.325147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.325131    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"default-dockercfg-hc4sm\""
Oct 31 15:04:44.325147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.325131    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"openshift-service-ca.crt\""
Oct 31 15:04:44.326956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.326941    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2348\"/\"kube-root-ca.crt\""
Oct 31 15:04:44.382421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.382339    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.383232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.383207    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.385849    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.385998    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.386061    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.386137    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.386167    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l847w\" (UniqueName: \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.386848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.386836    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.389815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.389790    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.389910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.389825    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.390616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.390600    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.447517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.447451    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l847w\" (UniqueName: \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\") pod \"distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") " pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:44.540146 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:44.540108    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:45.004062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:45.004030    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 15:04:45.004265 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:04:45.004244    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1859f7ae_117c_435f_8cf4_83d1d456e128.slice/crio-ee4818070ff7545851e6ffce1afc1e45a5ecb696c26d008b5f0f217ad386d532 WatchSource:0}: Error finding container ee4818070ff7545851e6ffce1afc1e45a5ecb696c26d008b5f0f217ad386d532: Status 404 returned error can't find the container with id ee4818070ff7545851e6ffce1afc1e45a5ecb696c26d008b5f0f217ad386d532
Oct 31 15:04:45.594009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:45.593956    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" event={"ID":"1859f7ae-117c-435f-8cf4-83d1d456e128","Type":"ContainerStarted","Data":"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"}
Oct 31 15:04:45.594484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:45.594027    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" event={"ID":"1859f7ae-117c-435f-8cf4-83d1d456e128","Type":"ContainerStarted","Data":"ee4818070ff7545851e6ffce1afc1e45a5ecb696c26d008b5f0f217ad386d532"}
Oct 31 15:04:45.710425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:45.710357    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podStartSLOduration=1.710339477 podStartE2EDuration="1.710339477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:45.709168946 +0000 UTC m=+1126.435449923" watchObservedRunningTime="2026-10-31 15:04:45.710339477 +0000 UTC m=+1126.436620457"
Oct 31 15:04:46.540302 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:46.540261    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:50.086597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:50.082243    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:50.384819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:50.384708    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:50.675649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:50.657711    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:50.762595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:50.754935    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:51.194598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.194506    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:51.225599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.196231    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:04:51.225599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.211206    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:04:51.654024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.653978    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:04:51.724603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.688114    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:51.741596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.730510    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:04:51.787348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:51.787321    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:04:54.541089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:54.541051    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:54.554702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:54.554676    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:04:54.812462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:04:54.812390    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:05:07.017531 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:05:07.017493    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:05:07.046588 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:07.046545    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"]
Oct 31 15:05:07.096011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:07.095975    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a975402d-jtsqc"]
Oct 31 15:05:07.422351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:07.422263    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4cc62b-c1f8-4752-8bc9-182a8225fcca" path="/var/lib/kubelet/pods/0a4cc62b-c1f8-4752-8bc9-182a8225fcca/volumes"
Oct 31 15:05:11.983599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:11.973118    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:05:12.000628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:11.997307    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.048960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.048919    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:05:12.084606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.082915    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\""
Oct 31 15:05:12.130715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.130668    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:12.212599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.208676    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:12.371602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.354233    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4mf\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.371990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.371959    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.379594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.375710    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.405714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.385154    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:12.406022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.405995    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.406240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.406220    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.406365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.406348    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.406539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.406523    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.435710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.435655    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.436018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.435997    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.554372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.554329    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.556389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.556365    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.557083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557062    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.557244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557227    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.557407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557389    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.557622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557601    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4mf\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.557853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557834    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.558011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.557994    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.558171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.558157    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.559247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.559226    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.559458 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.556293    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.559847 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.559826    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.563589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.560513    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.563760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.561115    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.567466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.567433    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.570457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.570423    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.595741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.595697    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:12.643645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.634144    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.643645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.634755    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4mf\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-926mt\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.656601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.654665    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:05:12.685599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.679015    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:12.685599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.679639    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:12.737607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.734667    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:12.817726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.817671    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:12.841017 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:12.831928    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:13.600403 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.599182    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:13.647610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.641697    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:13.767841 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.706661    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:05:13.767841 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.707881    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:13.767841 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.741668    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:13.801822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.801791    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:05:13.802079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.802064    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:05:13.802212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.802201    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:05:13.802334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.802322    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:05:13.802450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.802440    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:05:13.852650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.847596    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:13.852650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:13.848040    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:05:14.599793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.599743    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:14.698733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.673032    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:05:14.698733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.673129    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:05:14.698733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.673178    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:05:14.732046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.731057    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:14.733197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.733150    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:14.733541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.733523    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:05:14.743383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.742189    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:14.781182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.776114    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:14.781182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.776181    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:14.781182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.776316    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:14.814461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:14.814386    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.071599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.070997    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:15.095617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.075865    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.623789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.623733    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.624169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.624126    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.684727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.684681    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.757606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.751900    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.757606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.752312    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:15.757606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.752607    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:05:15.785657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.784946    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:05:15.814852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.814802    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:05:15.874693 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.874562    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:05:15.874994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:15.874957    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" event={"ID":"05994620-9cca-47b4-b56b-a9b76bd824e2","Type":"ContainerStarted","Data":"4d23c6e9e15668f8612d710b9cbe9d5c0075aa87b9a5a085e5254dfea2fb5186"}
Oct 31 15:05:16.079404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.078237    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:16.695600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.691664    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:16.776729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.776676    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:16.794275 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.793652    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:16.798597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.793693    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:16.798597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.793772    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:16.912611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.849586    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:05:16.938616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.922901    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:16.938616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:16.922966    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.034601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.030983    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" event={"ID":"05994620-9cca-47b4-b56b-a9b76bd824e2","Type":"ContainerStarted","Data":"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"}
Oct 31 15:05:17.089612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.084690    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.199830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.199765    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podStartSLOduration=6.199743413 podStartE2EDuration="6.199743413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:17.174621646 +0000 UTC m=+1157.900902625" watchObservedRunningTime="2026-10-31 15:05:17.199743413 +0000 UTC m=+1157.926024393"
Oct 31 15:05:17.684253 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.684219    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:17.684639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.684613    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:17.684816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.684793    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:17.693704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.693656    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.797756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.797699    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.810606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.798680    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.839606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.803722    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:17.840218 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.803777    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:17.918599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.907201    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Oct 31 15:05:17.918599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:17.907295    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:05:18.040599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.034702    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:18.040599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.034793    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:18.065600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.054738    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:18.065600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.054812    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:18.111661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.109866    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:18.685198 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.685156    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:18.685524 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.685470    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:18.717428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.716667    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:18.807769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.807716    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:05:18.850598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.832563    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:18.851239 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.838683    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:18.851447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.851415    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:18.852089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.852064    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:18.878220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.878180    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:18.921350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:18.921308    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:19.136608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.133868    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:19.253598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.249701    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6xfbz" podUID="057435af-b787-4b85-83e7-79af663f78ee" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.39:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:19.717599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.697971    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:19.717599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.698050    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:19.813606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.803153    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:19.813606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.809032    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:19.853924 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.853696    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:19.853924 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.853834    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:19.898098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.896311    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:05:19.923379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:19.923189    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:20.017279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.017248    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:05:20.047605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.044840    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:05:20.047605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.045841    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:05:20.344911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.342958    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:05:20.493537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.492523    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:05:20.564936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.564912    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:05:20.682925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.681999    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:20.682925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.682080    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:20.872658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.869661    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:20.942867 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:20.942783    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:05:21.680606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:21.680393    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:21.680606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:21.680475    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:22.681133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:22.681089    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:22.681530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:22.681161    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:23.681241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:23.681208    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:23.681681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:23.681261    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:24.681224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:24.681187    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:24.681402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:24.681243    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:25.680631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:25.680563    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:25.680820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:25.680656    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:26.681237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:26.681198    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 15:05:26.681665 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:26.681268    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 15:05:27.680627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:27.680594    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:27.684882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:27.684859    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:28.637618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:28.637582    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:05:37.194347 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:05:37.194283    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:05:38.764491 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:38.764456    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:38.780081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:38.780033    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:38.780671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:38.780640    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:38.783627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:38.783597    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:05:38.904461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:38.904403    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.076786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.076722    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.632594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.606269    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.806641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.791865    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.806641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.795401    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:05:39.812236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.812030    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.812236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.812117    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:39.812236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.812164    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:39.896813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.877080    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:05:39.896813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.877144    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:05:39.896813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.877180    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:05:39.911634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.909116    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:39.911634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:39.910309    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:05:40.094663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.094616    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:40.095605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.095554    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:05:40.296207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.296169    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x6jcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:05:40.296552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.296521    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:40.606676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.606431    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:40.611789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.611763    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:05:40.799812 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.799769    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:05:40.891297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.891193    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:40.902609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.894208    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:05:40.904224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:40.904176    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:41.031582 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.031516    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:41.095775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.095732    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:41.609274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.609222    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:41.806723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.806672    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:41.829726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.829678    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Oct 31 15:05:41.895725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.895613    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:41.907785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.907727    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:41.908622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:41.908592    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.043309 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.042495    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.096718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.096670    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.098616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.098405    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.190724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.190624    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.612783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.612734    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:42.835592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.808719    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.898417 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.898295    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.923536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.923495    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:42.925047 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:42.924343    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:43.043857 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.043666    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:43.125311 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.100676    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:05:43.280160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.280132    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:05:43.322200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.322169    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:05:43.617439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.617336    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:05:43.834262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.827898    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:05:43.834262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.831426    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:05:43.911659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.905457    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:05:43.973788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:43.973733    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:05:44.110728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:44.110678    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:05:44.304082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:44.304053    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:05:44.418597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:44.382736    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:05:44.798923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:44.798894    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:05:44.801245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:05:44.801222    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:06:00.335914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:00.335845    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:00.734805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:00.734697    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:00.895718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:00.895662    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:00.938073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:00.938022    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:06:00.938474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:00.938336    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:01.106793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:01.106729    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:01.720308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:01.720275    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:06:01.773474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:01.773433    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:01.780333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:01.780161    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:06:02.111637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:02.107668    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:02.111637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:02.109866    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:06:02.225215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:02.225182    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:06:02.359648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:02.344284    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:06:02.596207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:02.596176    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:06:07.013065 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:06:07.013026    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:06:12.422653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.422001    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:06:12.435685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.427394    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.489953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.489906    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.489959    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490015    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490095    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490163    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5zn\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490197    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490240    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490303    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.490523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.490328    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.586598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.538842    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.631361    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705182    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705246    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705338    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705367    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705442    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705556    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705682    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5zn\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705727    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.705779    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.706176    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.706850    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.710644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.707103    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:12.735593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:12.733266    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.071600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.068231    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.104603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.102213    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5zn\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.143861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.143826    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.144954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.144926    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-mklf2\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.328762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.328655    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:13.346700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.346655    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:13.409848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.409813    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:13.662823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.662713    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:13.838597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.837662    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:13.916672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:13.915666    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:14.006758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.006712    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:14.370601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.357175    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:14.371846 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.371821    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:06:14.484986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.484945    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:14.485596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.485542    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:06:14.847097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.846848    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:14.847552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.847526    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:06:14.857668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.857622    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:06:14.858789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.858749    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:06:14.858980 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.858963    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:06:14.859114 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.859099    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:06:15.097655 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:14.958275    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:06:15.100656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:15.041176    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:06:15.208690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:15.208616    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" event={"ID":"2c99068e-c7ce-48ba-a48c-07191442ca11","Type":"ContainerStarted","Data":"d8072268534a05beb2c7c8120b0f345f494dc1c41a9e48305bde722a4f2152a5"}
Oct 31 15:06:15.233970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:15.233945    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:06:15.386793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:15.384399    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:06:15.990217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:15.990190    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:06:16.244794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:16.244706    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" event={"ID":"2c99068e-c7ce-48ba-a48c-07191442ca11","Type":"ContainerStarted","Data":"b0db581002682e3663ada0463f81911803ab625b0d3aa6689a9af598b626b61c"}
Oct 31 15:06:16.413780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:16.413753    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:16.414915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:16.414867    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podStartSLOduration=4.414848421 podStartE2EDuration="4.414848421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:06:16.392520946 +0000 UTC m=+1217.118801924" watchObservedRunningTime="2026-10-31 15:06:16.414848421 +0000 UTC m=+1217.141129391"
Oct 31 15:06:16.415450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:16.415421    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:16.415705 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:16.415652    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:17.278177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.278131    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:06:17.475598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.475173    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:17.475598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.475236    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:17.913723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.913669    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:17.914463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.913994    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:17.933749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:17.933700    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:18.417990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:18.417951    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:18.418621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:18.418562    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:18.927783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:18.927712    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:18.928820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:18.928797    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:06:19.000201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.000151    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:19.042730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.042687    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:19.180160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.170170    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:06:19.244615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.236712    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:19.425980 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.411129    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:19.441728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:19.441631    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:20.422636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:20.417091    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:20.422636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:20.417153    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:21.422833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:21.422796    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:21.423421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:21.423387    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:22.420390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:22.420350    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:22.420714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:22.420684    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:23.410499 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:23.410465    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-mklf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Oct 31 15:06:23.410925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:23.410517    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Oct 31 15:06:24.855989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:24.855952    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:24.856623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:24.856598    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:24.857444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:24.857424    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:06:37.189607 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:06:37.187223    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:06:37.948602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:37.931920    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:37.997648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:37.988708    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:38.049534 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:38.048697    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:38.056551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:38.056524    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-x27md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:06:38.056835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:38.056810    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:38.463799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:38.463767    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-jrgbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:06:38.464252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:38.463841    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:58.921733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:58.921669    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:58.936301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:58.936198    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:06:59.050596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.049478    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:59.246049 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.245936    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:59.665600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.665182    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:06:59.948423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.948308    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:06:59.955323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.955294    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:06:59.988671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.964114    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:06:59.990107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:06:59.989241    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:07:00.044594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.015418    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.072956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.071700    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.072956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.072229    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:07:00.184357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.177909    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.264445 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.253895    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.264445 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.262771    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:07:00.698767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.698718    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.742599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.708102    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:07:00.966711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.961967    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:00.998507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:00.998463    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:01.078823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.078788    4549 scope.go:122] "RemoveContainer" containerID="c22f0954ef887c9f0b440ea2c2b9f93eeb15889c6d442e8606d13e5b8082b2be"
Oct 31 15:07:01.202612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.200758    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:01.235775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.235649    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:01.301645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.283329    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:07:01.582676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.531755    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:01.817592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.816692    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:07:01.980387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:01.980338    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:02.052599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.014334    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:02.089424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.027390    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:07:02.219037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.218990    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:02.265598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.240097    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:02.393805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.393774    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:07:02.538719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.538672    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:02.930144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.870894    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:07:02.930144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.916958    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:07:02.930144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:02.917050    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:03.101878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.101833    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:07:03.102827 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.102791    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:03.103314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.103293    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:07:03.266782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.264663    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:03.266782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.266355    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:03.549332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.549233    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:03.907655 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:03.878755    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:07:04.000781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.000721    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:07:04.142780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.109512    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:04.156989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.113717    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:07:04.180656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.157237    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:04.181449 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.141355    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:07:04.267751 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.267689    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:07:04.270598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.268375    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:07:04.556801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.556742    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:04.895692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:04.895019    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:07:05.131496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.130919    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.175656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.151960    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.270735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.205124    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:07:05.270735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.200890    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.270735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.200942    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.315338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.305672    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.315338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.306016    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:07:05.562657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.556968    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.581597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.577922    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.617617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.591616    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:05.633906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.631318    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:07:05.660962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.660935    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:07:05.945796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:05.945695    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:06.210137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.210009    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:06.210673 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.210637    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:07:06.340596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.339729    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:06.648607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.621718    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:07:06.790543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.790511    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:07:06.869598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.865225    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:07:06.991297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:06.968243    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:07:07.005835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.005805    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:07:07.070864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.070822    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:07:07.147389 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:07.147353    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:07:07.617879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.617834    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 15:07:07.624659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.624636    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.674054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.674024    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 15:07:07.677075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.677048    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"openshift-service-ca.crt\""
Oct 31 15:07:07.677075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.677048    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"default-dockercfg-zxkjx\""
Oct 31 15:07:07.677304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.677230    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"kube-root-ca.crt\""
Oct 31 15:07:07.677304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.677289    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"distinct-2-w0-56bc96-kserve-self-signed-certs\""
Oct 31 15:07:07.729349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729318    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9m6\" (UniqueName: \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.729528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729359    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.729528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729479    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.729700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729635    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.729810 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729791    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.729873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.729839    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.830947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.830911    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831145 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.830975    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831145 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831052    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9m6\" (UniqueName: \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831145 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831094    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831165    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831261    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831447    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.831692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831673    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.832014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.831990    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.834523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.834499    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.834782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.834762    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.889203 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.889131    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9m6\" (UniqueName: \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\") pod \"distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") " pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:07.938188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:07.938153    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:08.343123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:08.343088    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 15:07:08.344699 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:07:08.344673    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod827ff8fc_d174_4f66_838f_044a9201652e.slice/crio-8adce29fff2795c946822a4a03c0d27c645da39233cd263f6cc7dadcd41249e6 WatchSource:0}: Error finding container 8adce29fff2795c946822a4a03c0d27c645da39233cd263f6cc7dadcd41249e6: Status 404 returned error can't find the container with id 8adce29fff2795c946822a4a03c0d27c645da39233cd263f6cc7dadcd41249e6
Oct 31 15:07:08.380209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:08.380178    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" event={"ID":"827ff8fc-d174-4f66-838f-044a9201652e","Type":"ContainerStarted","Data":"8adce29fff2795c946822a4a03c0d27c645da39233cd263f6cc7dadcd41249e6"}
Oct 31 15:07:09.391054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:09.391012    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" event={"ID":"827ff8fc-d174-4f66-838f-044a9201652e","Type":"ContainerStarted","Data":"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"}
Oct 31 15:07:09.519071 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:09.519008    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podStartSLOduration=2.518990557 podStartE2EDuration="2.518990557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:09.516716782 +0000 UTC m=+1270.242997765" watchObservedRunningTime="2026-10-31 15:07:09.518990557 +0000 UTC m=+1270.245271536"
Oct 31 15:07:09.938920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:09.938885    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:17.938821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:17.938780    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:17.951855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:17.951831    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:18.473894 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:18.473864    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:07:25.402587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.402543    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:07:25.407258 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.407242    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.459763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.459728    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5695787b-serving-cert\""
Oct 31 15:07:25.494681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.494644    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9w2t\" (UniqueName: \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.494878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.494695    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.495139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.495114    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.495257 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.495185    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.522390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.522353    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:07:25.596014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.595983    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.596014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.596025    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.596297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.596085    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9w2t\" (UniqueName: \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.596297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.596109    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.596297 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:25.596146    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-serving-cert: secret "maas-api-e2e-mt-5695787b-serving-cert" not found
Oct 31 15:07:25.596297 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:25.596232    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls podName:dad18e79-ee73-4f4c-bbfb-8c801d0613bd nodeName:}" failed. No retries permitted until 2026-10-31 15:07:26.096208741 +0000 UTC m=+1286.822489717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls") pod "maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" (UID: "dad18e79-ee73-4f4c-bbfb-8c801d0613bd") : secret "maas-api-e2e-mt-5695787b-serving-cert" not found
Oct 31 15:07:25.597267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.597197    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.599517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.599497    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:25.659152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:25.659079    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9w2t\" (UniqueName: \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:26.102627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:26.102597    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:26.105108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:26.105083    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:26.322727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:26.322684    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:26.712093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:26.712069    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:07:26.712468 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:07:26.712218    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad18e79_ee73_4f4c_bbfb_8c801d0613bd.slice/crio-77f430bff7ae61b6e59bd7cf8fd901f580363da9aae4353a95ebb89a1f2b602b WatchSource:0}: Error finding container 77f430bff7ae61b6e59bd7cf8fd901f580363da9aae4353a95ebb89a1f2b602b: Status 404 returned error can't find the container with id 77f430bff7ae61b6e59bd7cf8fd901f580363da9aae4353a95ebb89a1f2b602b
Oct 31 15:07:26.713906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:26.713886    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:07:27.531120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:27.531078    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" event={"ID":"dad18e79-ee73-4f4c-bbfb-8c801d0613bd","Type":"ContainerStarted","Data":"77f430bff7ae61b6e59bd7cf8fd901f580363da9aae4353a95ebb89a1f2b602b"}
Oct 31 15:07:28.541367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:28.541332    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" event={"ID":"dad18e79-ee73-4f4c-bbfb-8c801d0613bd","Type":"ContainerStarted","Data":"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"}
Oct 31 15:07:28.542229 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:28.541679    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:28.665411 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:28.665358    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" podStartSLOduration=2.691898812 podStartE2EDuration="3.665342929s" totalImagesPullingTime="973.444117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:28.66358994 +0000 UTC m=+1289.389870922" watchObservedRunningTime="2026-10-31 15:07:28.665342929 +0000 UTC m=+1289.391623906"
Oct 31 15:07:34.554392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:34.554362    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:07:37.018280 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:37.018232    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:07:38.933604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:38.933539    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:07:38.937916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:38.937899    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.044837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.044792    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:07:39.077667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.077630    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.077883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.077680    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vfx\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.077883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.077837    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.077896    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.077982    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.078050    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.078109    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.078308    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.078398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.078344    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180056    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180106    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vfx\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180174    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180236    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180277    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180351    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180376    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180491    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180515    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180706    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.180861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.180826    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.181180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.181145    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.181310 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.181294    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.181387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.181318    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.182982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.182960    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.183368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.183347    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.237418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.237345    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.237418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.237354    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vfx\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.251392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.251359    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:39.687916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.687814    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:07:39.688304 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:07:39.688046    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd930d91f_40e7_40b4_a2e5_43908704cbcd.slice/crio-877934fa0ed0567bbe962f982f15c3391e36083e0730bd303a8a0128dbfb1aa5 WatchSource:0}: Error finding container 877934fa0ed0567bbe962f982f15c3391e36083e0730bd303a8a0128dbfb1aa5: Status 404 returned error can't find the container with id 877934fa0ed0567bbe962f982f15c3391e36083e0730bd303a8a0128dbfb1aa5
Oct 31 15:07:39.690676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.690641    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:07:39.690775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.690706    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:07:39.690775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:39.690742    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:07:40.637728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:40.637639    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" event={"ID":"d930d91f-40e7-40b4-a2e5-43908704cbcd","Type":"ContainerStarted","Data":"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"}
Oct 31 15:07:40.637728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:40.637705    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" event={"ID":"d930d91f-40e7-40b4-a2e5-43908704cbcd","Type":"ContainerStarted","Data":"877934fa0ed0567bbe962f982f15c3391e36083e0730bd303a8a0128dbfb1aa5"}
Oct 31 15:07:40.861494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:40.861435    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podStartSLOduration=2.861418458 podStartE2EDuration="2.861418458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:40.811280092 +0000 UTC m=+1301.537561072" watchObservedRunningTime="2026-10-31 15:07:40.861418458 +0000 UTC m=+1301.587699436"
Oct 31 15:07:41.252418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:41.252378    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:41.254062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:41.254039    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:41.254137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:41.254093    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:42.252536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:42.252499    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:42.253052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:42.252554    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:43.252169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:43.252127    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:43.252353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:43.252187    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:44.251948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:44.251913    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:44.252364 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:44.251980    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:45.074411 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.074376    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:07:45.078508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.078488    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.133176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.133146    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c3356b-serving-cert\""
Oct 31 15:07:45.144535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.144501    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:07:45.184695 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.184656    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.184887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.184741    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.184887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.184849    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6czg\" (UniqueName: \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.184887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.184883    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.258116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.258058    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:45.258906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.258863    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:45.286759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.286730    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.286938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.286790    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.286938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.286857    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6czg\" (UniqueName: \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.286938 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:45.286871    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-serving-cert: secret "maas-api-e2e-api-a-c3356b-serving-cert" not found
Oct 31 15:07:45.286938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.286880    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.286938 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:07:45.286931    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls podName:9147ef9c-fc31-4a5c-946a-0f00bd254731 nodeName:}" failed. No retries permitted until 2026-10-31 15:07:45.786915908 +0000 UTC m=+1306.513196863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls") pod "maas-api-e2e-api-a-c3356b-589f6766b-9drhc" (UID: "9147ef9c-fc31-4a5c-946a-0f00bd254731") : secret "maas-api-e2e-api-a-c3356b-serving-cert" not found
Oct 31 15:07:45.287446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.287430    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.289494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.289475    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.359208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.359110    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6czg\" (UniqueName: \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.796614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.796560    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.799265 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.799235    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") pod \"maas-api-e2e-api-a-c3356b-589f6766b-9drhc\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:45.992493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:45.992464    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:46.251756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:46.251722    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 15:07:46.251933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:46.251776    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 15:07:46.387400 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:07:46.387365    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9147ef9c_fc31_4a5c_946a_0f00bd254731.slice/crio-d839f86ccdb5b66f7d629de6c82e13e8f46f96bcad3b7d2c865ef9dde246b52c WatchSource:0}: Error finding container d839f86ccdb5b66f7d629de6c82e13e8f46f96bcad3b7d2c865ef9dde246b52c: Status 404 returned error can't find the container with id d839f86ccdb5b66f7d629de6c82e13e8f46f96bcad3b7d2c865ef9dde246b52c
Oct 31 15:07:46.389032 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:46.389001    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:07:46.682656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:46.682539    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" event={"ID":"9147ef9c-fc31-4a5c-946a-0f00bd254731","Type":"ContainerStarted","Data":"d839f86ccdb5b66f7d629de6c82e13e8f46f96bcad3b7d2c865ef9dde246b52c"}
Oct 31 15:07:47.569177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.569144    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:47.570239 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.570210    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:47.572451 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.572399    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:07:47.691380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.691345    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" event={"ID":"9147ef9c-fc31-4a5c-946a-0f00bd254731","Type":"ContainerStarted","Data":"ab8f190db9811c17bb93ce6ef25a638cb8ca7d2b5a81a2cd7a152c2cfe2230a2"}
Oct 31 15:07:47.691613 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.691403    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:07:47.851435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:47.851318    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" podStartSLOduration=2.043754809 podStartE2EDuration="2.85129895s" totalImagesPullingTime="807.544141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:47.849024084 +0000 UTC m=+1308.575305086" watchObservedRunningTime="2026-10-31 15:07:47.85129895 +0000 UTC m=+1308.577579928"
Oct 31 15:07:53.706099 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:07:53.706065    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:08:07.014196 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:08:07.014158    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:08:16.333199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.333166    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:08:16.338761 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.338741    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.392040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.392003    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5695787b\"/\"openshift-service-ca.crt\""
Oct 31 15:08:16.396495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.396461    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:08:16.439225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.439193    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5695787b\"/\"kube-root-ca.crt\""
Oct 31 15:08:16.439225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.439206    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5695787b\"/\"e2e-lifecycle-model-5695787b-kserve-self-signed-certs\""
Oct 31 15:08:16.439453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.439248    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5695787b\"/\"default-dockercfg-b7k29\""
Oct 31 15:08:16.462437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462402    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.462620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462527    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.462620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462557    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.462771 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462753    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.462823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462814    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.462962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.462941    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjp98\" (UniqueName: \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.564859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.564807    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.564891    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.564966    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjp98\" (UniqueName: \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.565026    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.565106    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.565138    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.565852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.565826    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.566996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.566973    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.567646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.567323    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.570945 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.570923    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.578361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.578287    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.623787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.623695    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjp98\" (UniqueName: \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\") pod \"e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") " pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:16.654517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:16.654485    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:17.048149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:17.048121    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:08:17.048920 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:08:17.048887    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod469f9541_1893_4d0a_899a_698108d2c6e4.slice/crio-4e617528fea7ca53b8e66956a7bec9cee84608f6065cbb8c03d7764cd83beccc WatchSource:0}: Error finding container 4e617528fea7ca53b8e66956a7bec9cee84608f6065cbb8c03d7764cd83beccc: Status 404 returned error can't find the container with id 4e617528fea7ca53b8e66956a7bec9cee84608f6065cbb8c03d7764cd83beccc
Oct 31 15:08:18.015194 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:18.015150    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" event={"ID":"469f9541-1893-4d0a-899a-698108d2c6e4","Type":"ContainerStarted","Data":"48daf72589938ce1742197833bc920d0bc7804b68781159d9166a5f06bf801fb"}
Oct 31 15:08:18.015712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:18.015215    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" event={"ID":"469f9541-1893-4d0a-899a-698108d2c6e4","Type":"ContainerStarted","Data":"4e617528fea7ca53b8e66956a7bec9cee84608f6065cbb8c03d7764cd83beccc"}
Oct 31 15:08:18.135725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:18.135657    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podStartSLOduration=2.135637048 podStartE2EDuration="2.135637048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:18.131132319 +0000 UTC m=+1338.857413300" watchObservedRunningTime="2026-10-31 15:08:18.135637048 +0000 UTC m=+1338.861918031"
Oct 31 15:08:18.655324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:18.655293    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:26.655063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:26.655025    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:26.670370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:26.670339    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:27.091501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:27.091470    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:29.087120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.085706    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 15:08:29.114896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.113647    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:08:29.229713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.229667    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:29.275768 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.275724    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:29.484654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.471241    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 15:08:29.546599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.545776    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:29.804624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.804472    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:29.875979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:29.875934    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.123696 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.123541    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.125825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.124759    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:08:30.223721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.223676    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:08:30.233444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.233405    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.234024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.234004    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:08:30.314596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.314526    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.325590 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.325149    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:08:30.490781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.490680    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.491747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.491723    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:08:30.549429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.549371    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.551264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.551229    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:08:30.806897 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.806853    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:08:30.808756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.808730    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:08:30.903596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.893445    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:30.949598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:30.905126    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:08:31.124694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.124598    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:31.207625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.201115    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:08:31.220627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.215526    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:08:31.263406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.262420    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:31.413654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.413116    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:31.417733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.413883    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:08:31.529748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.529702    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:31.575940 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.574675    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:31.807672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:31.807611    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.013686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.004403    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:08:32.071347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.070816    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:08:32.119658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.106871    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.119658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.107462    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 15:08:32.160604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.156345    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.167237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.167044    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.228786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.228736    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.274608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.265233    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:08:32.274608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.266486    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.418993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.418861    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:32.821948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:32.821903    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:08:33.207657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.207077    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.226192 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.208152    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:33.229449 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.229075    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.230379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.230351    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.230851 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.230832    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:08:33.277720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.277670    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.295217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.281533    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.462731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.429833    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:08:33.835824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.835788    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:08:33.848627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.840662    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-4q2wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:08:33.848627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.840734    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:08:33.849599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.848961    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:08:33.892797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.890493    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:08:33.898810 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:33.897511    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:08:34.062838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:34.062734    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:08:34.063671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:34.063638    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:08:37.018183 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:08:37.018144    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:08:39.397725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.397696    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:08:39.402040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.402023    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.448672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.448632    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rnf\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.448857 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.448767    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.448997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.448969    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449100 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449013    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449185    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449227    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449445 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449423    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449468    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.449633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.449540    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.508812 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.508775    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:08:39.550897 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.550868    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rnf\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.550909    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551020    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551045    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551117    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551141    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551233    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551264    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551328    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.551952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.551927    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.552173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.552154    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.552276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.552177    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.552440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.552410    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.552973 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.552950    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.554381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.554356    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.555164 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.555141    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.608151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.608123    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.608290 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.608155    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rnf\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:39.716138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:39.716044    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:40.150913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.150882    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:08:40.151495 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:08:40.151470    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6824f94f_203d_4f4b_84ba_a9c6a81f736a.slice/crio-2f0f59abae742eb2bd2d883619c0dbe68ff1fbad7a99406a6ba1fa17f1515f99 WatchSource:0}: Error finding container 2f0f59abae742eb2bd2d883619c0dbe68ff1fbad7a99406a6ba1fa17f1515f99: Status 404 returned error can't find the container with id 2f0f59abae742eb2bd2d883619c0dbe68ff1fbad7a99406a6ba1fa17f1515f99
Oct 31 15:08:40.154191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.154156    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:08:40.154285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.154240    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:08:40.154353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.154288    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:08:40.530413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.530369    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" event={"ID":"6824f94f-203d-4f4b-84ba-a9c6a81f736a","Type":"ContainerStarted","Data":"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"}
Oct 31 15:08:40.530882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.530437    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" event={"ID":"6824f94f-203d-4f4b-84ba-a9c6a81f736a","Type":"ContainerStarted","Data":"2f0f59abae742eb2bd2d883619c0dbe68ff1fbad7a99406a6ba1fa17f1515f99"}
Oct 31 15:08:40.655460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.655282    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podStartSLOduration=1.6552619549999998 podStartE2EDuration="1.655261955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:40.649746922 +0000 UTC m=+1361.376027900" watchObservedRunningTime="2026-10-31 15:08:40.655261955 +0000 UTC m=+1361.381542935"
Oct 31 15:08:40.723749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:40.723713    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:41.721479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:41.721425    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:41.722128 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:41.722097    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:42.730993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:42.730958    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:42.731638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:42.731605    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:43.724511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:43.722721    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:43.724511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:43.722810    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:44.723600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:44.719967    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:44.723600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:44.720030    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:45.720455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:45.720419    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:45.720751 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:45.720723    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:46.716612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:46.716563    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Oct 31 15:08:46.716999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:46.716653    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Oct 31 15:08:48.095584 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:48.095540    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:48.095985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:48.095949    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:08:48.096684 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:08:48.096664    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:09:06.163598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.163542    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:09:06.167786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.167769    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.224624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.224591    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:09:06.269627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.269592    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4d3ea8-serving-cert\""
Oct 31 15:09:06.274487 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.274458    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.274668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.274648    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdnwl\" (UniqueName: \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.274722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.274681    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.274788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.274776    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.375653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.375618    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdnwl\" (UniqueName: \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.375653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.375666    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.375941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.375864    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.376007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.375979    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.376081 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:06.376058    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-serving-cert: secret "maas-api-e2e-api-a-4d3ea8-serving-cert" not found
Oct 31 15:09:06.376146 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:06.376133    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls podName:0212fd24-d483-41cb-affa-c144195698c6 nodeName:}" failed. No retries permitted until 2026-10-31 15:09:06.876114187 +0000 UTC m=+1387.602395146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls") pod "maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" (UID: "0212fd24-d483-41cb-affa-c144195698c6") : secret "maas-api-e2e-api-a-4d3ea8-serving-cert" not found
Oct 31 15:09:06.376234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.376218    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.378453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.378433    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.434884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.434801    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdnwl\" (UniqueName: \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.884715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.884677    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:06.887361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:06.887301    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") pod \"maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:07.013605 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:07.013541    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:09:07.089864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:07.089825    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:07.566133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:07.563292    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:09:07.760637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:07.760602    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" event={"ID":"0212fd24-d483-41cb-affa-c144195698c6","Type":"ContainerStarted","Data":"33976ab76bad88c53fb44428497a00e29f7f9c981dec401ddf078b2432bfd0dd"}
Oct 31 15:09:08.769561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:08.769526    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" event={"ID":"0212fd24-d483-41cb-affa-c144195698c6","Type":"ContainerStarted","Data":"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"}
Oct 31 15:09:08.769979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:08.769622    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:08.886089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:08.886034    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" podStartSLOduration=1.9839528770000001 podStartE2EDuration="2.886018345s" totalImagesPullingTime="902.065468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:08.883273014 +0000 UTC m=+1389.609553992" watchObservedRunningTime="2026-10-31 15:09:08.886018345 +0000 UTC m=+1389.612299323"
Oct 31 15:09:14.782275 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:14.782197    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:09:26.598597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:26.595747    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:09:26.906656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:26.905714    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:09:27.066603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:27.066529    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:09:27.067235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:27.067206    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded"
Oct 31 15:09:37.014811 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:37.014761    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:09:45.943654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:45.943611    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-f284c1"]
Oct 31 15:09:45.947946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:45.947928    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:45.997267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:45.997236    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq497\" (UniqueName: \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\") pod \"test-curl-7585-f284c1\" (UID: \"faeecac1-0b5a-42d6-9949-4ec89f10cc57\") " pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:46.004270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.004238    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-f284c1"]
Oct 31 15:09:46.048523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.048491    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\""
Oct 31 15:09:46.098274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.098235    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq497\" (UniqueName: \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\") pod \"test-curl-7585-f284c1\" (UID: \"faeecac1-0b5a-42d6-9949-4ec89f10cc57\") " pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:46.155383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.155345    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq497\" (UniqueName: \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\") pod \"test-curl-7585-f284c1\" (UID: \"faeecac1-0b5a-42d6-9949-4ec89f10cc57\") " pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:46.260450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.260364    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:46.639364 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:46.639336    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-f284c1"]
Oct 31 15:09:46.640113 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:09:46.640083    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaeecac1_0b5a_42d6_9949_4ec89f10cc57.slice/crio-cf97edefefb3ebe2ccae2687e55eb496e34996a05e2dafa2c5bc711b45d47223 WatchSource:0}: Error finding container cf97edefefb3ebe2ccae2687e55eb496e34996a05e2dafa2c5bc711b45d47223: Status 404 returned error can't find the container with id cf97edefefb3ebe2ccae2687e55eb496e34996a05e2dafa2c5bc711b45d47223
Oct 31 15:09:47.155679 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:47.155636    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-f284c1" event={"ID":"faeecac1-0b5a-42d6-9949-4ec89f10cc57","Type":"ContainerStarted","Data":"cf97edefefb3ebe2ccae2687e55eb496e34996a05e2dafa2c5bc711b45d47223"}
Oct 31 15:09:50.184444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:50.184408    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-f284c1" event={"ID":"faeecac1-0b5a-42d6-9949-4ec89f10cc57","Type":"ContainerStarted","Data":"2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e"}
Oct 31 15:09:50.299976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:50.299922    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7585-f284c1" podStartSLOduration=1.9497124719999999 podStartE2EDuration="5.299906946s" totalImagesPullingTime="3.350194474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:50.295971331 +0000 UTC m=+1431.022252309" watchObservedRunningTime="2026-10-31 15:09:50.299906946 +0000 UTC m=+1431.026187924"
Oct 31 15:09:51.775423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:51.775380    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-f284c1"]
Oct 31 15:09:51.780303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:51.780275    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-f284c1"]
Oct 31 15:09:52.198360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:52.198271    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7585-f284c1" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" containerName="test-curl-7585-f284c1" containerID="cri-o://2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e" gracePeriod=2
Oct 31 15:09:52.249075 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:52.249026    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-f284c1\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:54.463463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:54.463404    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:54.527095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:54.527060    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\" (UniqueName: \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\") pod \"faeecac1-0b5a-42d6-9949-4ec89f10cc57\" (UID: \"faeecac1-0b5a-42d6-9949-4ec89f10cc57\") "
Oct 31 15:09:54.529423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:54.529395    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497" pod "faeecac1-0b5a-42d6-9949-4ec89f10cc57" (UID: "faeecac1-0b5a-42d6-9949-4ec89f10cc57"). InnerVolumeSpecName "kube-api-access-tq497". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:09:54.628547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:54.628505    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq497\" (UniqueName: \"kubernetes.io/projected/faeecac1-0b5a-42d6-9949-4ec89f10cc57-kube-api-access-tq497\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:09:55.236398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.236358    4549 generic.go:356] "Generic (PLEG): container finished" podID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" containerID="2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e" exitCode=137
Oct 31 15:09:55.236398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.236401    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:55.236658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.236417    4549 scope.go:122] "RemoveContainer" containerID="2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e"
Oct 31 15:09:55.253564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.253544    4549 scope.go:122] "RemoveContainer" containerID="2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e"
Oct 31 15:09:55.254013 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:55.253987    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e\": container with ID starting with 2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e not found: ID does not exist" containerID="2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e"
Oct 31 15:09:55.254102 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.254026    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e"} err="failed to get container status \"2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e\": rpc error: code = NotFound desc = could not find container \"2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e\": container with ID starting with 2d31c2ac1139d86127c1fe3b4f054c08eea99ac4142bc1c2229084de399ce47e not found: ID does not exist"
Oct 31 15:09:55.301944 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:09:55.301904    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-f284c1\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" pod="opendatahub/test-curl-7585-f284c1"
Oct 31 15:09:55.421142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:09:55.421107    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" path="/var/lib/kubelet/pods/faeecac1-0b5a-42d6-9949-4ec89f10cc57/volumes"
Oct 31 15:10:06.352908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.352871    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:10:06.353796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.353775    4549 cpu_manager.go:414] "containerMap: removing container" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" containerName="test-curl-7585-f284c1"
Oct 31 15:10:06.353866 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.353799    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" containerName="test-curl-7585-f284c1"
Oct 31 15:10:06.353958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.353947    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeecac1-0b5a-42d6-9949-4ec89f10cc57" containerName="test-curl-7585-f284c1"
Oct 31 15:10:06.359349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.359331    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.418344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.418250    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5f17a1-openshift-default-dockercfg-96fx9\""
Oct 31 15:10:06.434591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.430376    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:10:06.465488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.465446    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2hz\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.465719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.465636    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.465806 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.465776    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.465950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.465935    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.466022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.465963    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.466080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.466043    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.466208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.466190    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.466305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.466288    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.466387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.466372    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.608173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.607884    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.610056 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.609912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.610327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.609986    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.610500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.610376    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.610640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.610620    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.610849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.610833    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.611053 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.610871    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.611202 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.611185    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.611324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.611310    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.612474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.612453    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2hz\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.616078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.616054    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.616749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.616729    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.617013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.612175    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.617132 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.611474    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.619917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.619894    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.627627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.625138    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.680611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.679822    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2hz\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.685604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.683349    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:06.973698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:06.973606    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:07.023800 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:07.021450    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:10:07.422539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:07.422512    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:10:07.423910 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:10:07.423881    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4635b899_8973_4181_afa5_c45aa5471013.slice/crio-c13e2c7bf3566bd96892da3fec0d978afb474848141cf17e43ac4dcdd01dc58b WatchSource:0}: Error finding container c13e2c7bf3566bd96892da3fec0d978afb474848141cf17e43ac4dcdd01dc58b: Status 404 returned error can't find the container with id c13e2c7bf3566bd96892da3fec0d978afb474848141cf17e43ac4dcdd01dc58b
Oct 31 15:10:07.427027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:07.426985    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:10:07.427115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:07.427076    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:10:07.427165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:07.427121    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:10:08.359155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:08.359122    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerStarted","Data":"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"}
Oct 31 15:10:08.359352 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:08.359184    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerStarted","Data":"c13e2c7bf3566bd96892da3fec0d978afb474848141cf17e43ac4dcdd01dc58b"}
Oct 31 15:10:08.525439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:08.525376    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podStartSLOduration=2.525354736 podStartE2EDuration="2.525354736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:08.52424293 +0000 UTC m=+1449.250523922" watchObservedRunningTime="2026-10-31 15:10:08.525354736 +0000 UTC m=+1449.251635821"
Oct 31 15:10:08.974635 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:08.974596    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:08.980483 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:08.980459    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:09.365728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:09.365696    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:09.366656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:09.366637    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:10:37.018632 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:37.018591    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:10:40.384189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.384155    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:10:40.391844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.391825    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.444499 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.444468    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:10:40.489121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.489064    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-aac61b-serving-cert\""
Oct 31 15:10:40.566531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.566499    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9xv\" (UniqueName: \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.566741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.566542    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.566790 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.566773    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.566966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.566939    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.667984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.667891    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9xv\" (UniqueName: \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.667984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.667930    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.668213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.668043    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.668213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.668111    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.668335 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:40.668264    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-serving-cert: secret "maas-api-e2e-api-a-aac61b-serving-cert" not found
Oct 31 15:10:40.668390 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:40.668352    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls podName:ccfe8492-51bd-4a3e-aff9-255dbecee653 nodeName:}" failed. No retries permitted until 2026-10-31 15:10:41.168330554 +0000 UTC m=+1481.894611515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls") pod "maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" (UID: "ccfe8492-51bd-4a3e-aff9-255dbecee653") : secret "maas-api-e2e-api-a-aac61b-serving-cert" not found
Oct 31 15:10:40.668736 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.668714    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.670773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.670747    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:40.726365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:40.726329    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9xv\" (UniqueName: \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:41.175671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:41.175629    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:41.178175 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:41.178156    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") pod \"maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:41.307246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:41.307208    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:41.684472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:41.684440    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:10:41.685089 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:10:41.685068    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfe8492_51bd_4a3e_aff9_255dbecee653.slice/crio-00614804dd2d94e8258ee902724daa58d1df7511ecf1053bf7c7d46b5da4f7c5 WatchSource:0}: Error finding container 00614804dd2d94e8258ee902724daa58d1df7511ecf1053bf7c7d46b5da4f7c5: Status 404 returned error can't find the container with id 00614804dd2d94e8258ee902724daa58d1df7511ecf1053bf7c7d46b5da4f7c5
Oct 31 15:10:42.673153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:42.673117    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" event={"ID":"ccfe8492-51bd-4a3e-aff9-255dbecee653","Type":"ContainerStarted","Data":"00614804dd2d94e8258ee902724daa58d1df7511ecf1053bf7c7d46b5da4f7c5"}
Oct 31 15:10:43.682732 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:43.682692    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" event={"ID":"ccfe8492-51bd-4a3e-aff9-255dbecee653","Type":"ContainerStarted","Data":"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"}
Oct 31 15:10:43.683187 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:43.682872    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:43.799981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:43.799929    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" podStartSLOduration=2.734491306 podStartE2EDuration="3.799914341s" totalImagesPullingTime="1.065423035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:43.795445325 +0000 UTC m=+1484.521726315" watchObservedRunningTime="2026-10-31 15:10:43.799914341 +0000 UTC m=+1484.526195340"
Oct 31 15:10:49.524553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.524521    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:10:49.524990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.524750    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" podUID="9147ef9c-fc31-4a5c-946a-0f00bd254731" containerName="maas-api" containerID="cri-o://ab8f190db9811c17bb93ce6ef25a638cb8ca7d2b5a81a2cd7a152c2cfe2230a2" gracePeriod=30
Oct 31 15:10:49.696642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.696610    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:10:49.737107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.737071    4549 generic.go:356] "Generic (PLEG): container finished" podID="9147ef9c-fc31-4a5c-946a-0f00bd254731" containerID="ab8f190db9811c17bb93ce6ef25a638cb8ca7d2b5a81a2cd7a152c2cfe2230a2" exitCode=0
Oct 31 15:10:49.737628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.737595    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" event={"ID":"9147ef9c-fc31-4a5c-946a-0f00bd254731","Type":"ContainerDied","Data":"ab8f190db9811c17bb93ce6ef25a638cb8ca7d2b5a81a2cd7a152c2cfe2230a2"}
Oct 31 15:10:49.992667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:49.992639    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:10:50.114211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.114179    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\") pod \"9147ef9c-fc31-4a5c-946a-0f00bd254731\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") "
Oct 31 15:10:50.114813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.114787    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") pod \"9147ef9c-fc31-4a5c-946a-0f00bd254731\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") "
Oct 31 15:10:50.115334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.115313    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\" (UniqueName: \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\") pod \"9147ef9c-fc31-4a5c-946a-0f00bd254731\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") "
Oct 31 15:10:50.115523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.115507    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\") pod \"9147ef9c-fc31-4a5c-946a-0f00bd254731\" (UID: \"9147ef9c-fc31-4a5c-946a-0f00bd254731\") "
Oct 31 15:10:50.116654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.115157    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca" pod "9147ef9c-fc31-4a5c-946a-0f00bd254731" (UID: "9147ef9c-fc31-4a5c-946a-0f00bd254731"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:10:50.126415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.126358    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls" pod "9147ef9c-fc31-4a5c-946a-0f00bd254731" (UID: "9147ef9c-fc31-4a5c-946a-0f00bd254731"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:10:50.127629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.127250    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs" pod "9147ef9c-fc31-4a5c-946a-0f00bd254731" (UID: "9147ef9c-fc31-4a5c-946a-0f00bd254731"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:10:50.127943 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.127916    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg" pod "9147ef9c-fc31-4a5c-946a-0f00bd254731" (UID: "9147ef9c-fc31-4a5c-946a-0f00bd254731"). InnerVolumeSpecName "kube-api-access-t6czg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:10:50.224331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.224239    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9147ef9c-fc31-4a5c-946a-0f00bd254731-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:10:50.224331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.224273    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:10:50.224331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.224286    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6czg\" (UniqueName: \"kubernetes.io/projected/9147ef9c-fc31-4a5c-946a-0f00bd254731-kube-api-access-t6czg\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:10:50.224331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.224298    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9147ef9c-fc31-4a5c-946a-0f00bd254731-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:10:50.748029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.747993    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc" event={"ID":"9147ef9c-fc31-4a5c-946a-0f00bd254731","Type":"ContainerDied","Data":"d839f86ccdb5b66f7d629de6c82e13e8f46f96bcad3b7d2c865ef9dde246b52c"}
Oct 31 15:10:50.748029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.748016    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"
Oct 31 15:10:50.748525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.748061    4549 scope.go:122] "RemoveContainer" containerID="ab8f190db9811c17bb93ce6ef25a638cb8ca7d2b5a81a2cd7a152c2cfe2230a2"
Oct 31 15:10:50.913592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.913543    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:10:50.963400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:50.963370    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3356b-589f6766b-9drhc"]
Oct 31 15:10:51.421803 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:51.421768    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9147ef9c-fc31-4a5c-946a-0f00bd254731" path="/var/lib/kubelet/pods/9147ef9c-fc31-4a5c-946a-0f00bd254731/volumes"
Oct 31 15:10:55.861161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.861127    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:10:55.861740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.861726    4549 cpu_manager.go:414] "containerMap: removing container" podUID="9147ef9c-fc31-4a5c-946a-0f00bd254731" containerName="maas-api"
Oct 31 15:10:55.861784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.861740    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9147ef9c-fc31-4a5c-946a-0f00bd254731" containerName="maas-api"
Oct 31 15:10:55.861868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.861860    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9147ef9c-fc31-4a5c-946a-0f00bd254731" containerName="maas-api"
Oct 31 15:10:55.866555 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.866538    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:55.924116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:55.924084    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:10:56.006643 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.006547    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.008858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.008833    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqsdj\" (UniqueName: \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.009228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.009206    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.009453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.009439    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.021339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.021313    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5f17a1-serving-cert\""
Oct 31 15:10:56.111332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.111231    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.111332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.111282    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.111605 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:56.111388    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-serving-cert: secret "maas-api-e2e-api-b-5f17a1-serving-cert" not found
Oct 31 15:10:56.111605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.111450    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.111605 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:10:56.111467    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls podName:929a3d31-fff6-4e55-bdcb-7e04566d6dca nodeName:}" failed. No retries permitted until 2026-10-31 15:10:56.611445665 +0000 UTC m=+1497.337726643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls") pod "maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" (UID: "929a3d31-fff6-4e55-bdcb-7e04566d6dca") : secret "maas-api-e2e-api-b-5f17a1-serving-cert" not found
Oct 31 15:10:56.111605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.111589    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqsdj\" (UniqueName: \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.112187 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.112166    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.114434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.114414    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.169977 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.169945    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqsdj\" (UniqueName: \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.619680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.619637    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.622376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.622347    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") pod \"maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:56.780326 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:56.780289    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:57.165501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:57.165472    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:10:57.167870 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:10:57.167846    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929a3d31_fff6_4e55_bdcb_7e04566d6dca.slice/crio-3f7c1eb3e48ddd97d2187ad563f4fcc94a117496dd2e04ff2ec46a3cab5e2dcd WatchSource:0}: Error finding container 3f7c1eb3e48ddd97d2187ad563f4fcc94a117496dd2e04ff2ec46a3cab5e2dcd: Status 404 returned error can't find the container with id 3f7c1eb3e48ddd97d2187ad563f4fcc94a117496dd2e04ff2ec46a3cab5e2dcd
Oct 31 15:10:57.821635 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:57.821542    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" event={"ID":"929a3d31-fff6-4e55-bdcb-7e04566d6dca","Type":"ContainerStarted","Data":"3f7c1eb3e48ddd97d2187ad563f4fcc94a117496dd2e04ff2ec46a3cab5e2dcd"}
Oct 31 15:10:58.835610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:58.835540    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" event={"ID":"929a3d31-fff6-4e55-bdcb-7e04566d6dca","Type":"ContainerStarted","Data":"67c5bd219bc57d98a73b0e438f6a0eb7aa2002fadcb376caaf62f456e8d00a88"}
Oct 31 15:10:58.836115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:58.835656    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:10:58.955753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:10:58.955685    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" podStartSLOduration=3.162558134 podStartE2EDuration="3.95566556s" totalImagesPullingTime="793.107426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:58.950077714 +0000 UTC m=+1499.676358695" watchObservedRunningTime="2026-10-31 15:10:58.95566556 +0000 UTC m=+1499.681946540"
Oct 31 15:11:02.682135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:02.682106    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:11:02.790641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:02.790618    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:11:04.853430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:04.853398    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:11:07.014337 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:11:07.014297    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:11:20.145580 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.145541    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:11:20.146160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.145765    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" podUID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" containerName="maas-api" containerID="cri-o://8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a" gracePeriod=30
Oct 31 15:11:20.560141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.560109    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:11:20.608732 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.608698    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") pod \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") "
Oct 31 15:11:20.608943 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.608777    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\") pod \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") "
Oct 31 15:11:20.608943 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.608805    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\") pod \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") "
Oct 31 15:11:20.609067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.608941    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\" (UniqueName: \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\") pod \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\" (UID: \"dad18e79-ee73-4f4c-bbfb-8c801d0613bd\") "
Oct 31 15:11:20.610179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.610150    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca" pod "dad18e79-ee73-4f4c-bbfb-8c801d0613bd" (UID: "dad18e79-ee73-4f4c-bbfb-8c801d0613bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:11:20.622628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.615181    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t" pod "dad18e79-ee73-4f4c-bbfb-8c801d0613bd" (UID: "dad18e79-ee73-4f4c-bbfb-8c801d0613bd"). InnerVolumeSpecName "kube-api-access-k9w2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:11:20.623105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.623073    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs" pod "dad18e79-ee73-4f4c-bbfb-8c801d0613bd" (UID: "dad18e79-ee73-4f4c-bbfb-8c801d0613bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:11:20.626769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.626716    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls" pod "dad18e79-ee73-4f4c-bbfb-8c801d0613bd" (UID: "dad18e79-ee73-4f4c-bbfb-8c801d0613bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:11:20.710838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.710805    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:20.710838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.710840    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:20.711066 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.710858    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:20.711066 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:20.710873    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9w2t\" (UniqueName: \"kubernetes.io/projected/dad18e79-ee73-4f4c-bbfb-8c801d0613bd-kube-api-access-k9w2t\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:21.014303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.014264    4549 generic.go:356] "Generic (PLEG): container finished" podID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" containerID="8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a" exitCode=0
Oct 31 15:11:21.014513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.014316    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"
Oct 31 15:11:21.014513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.014340    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" event={"ID":"dad18e79-ee73-4f4c-bbfb-8c801d0613bd","Type":"ContainerDied","Data":"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"}
Oct 31 15:11:21.014513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.014408    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6" event={"ID":"dad18e79-ee73-4f4c-bbfb-8c801d0613bd","Type":"ContainerDied","Data":"77f430bff7ae61b6e59bd7cf8fd901f580363da9aae4353a95ebb89a1f2b602b"}
Oct 31 15:11:21.014513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.014432    4549 scope.go:122] "RemoveContainer" containerID="8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"
Oct 31 15:11:21.028941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.028914    4549 scope.go:122] "RemoveContainer" containerID="8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"
Oct 31 15:11:21.029282 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:11:21.029261    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a\": container with ID starting with 8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a not found: ID does not exist" containerID="8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"
Oct 31 15:11:21.029339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.029292    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a"} err="failed to get container status \"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a\": rpc error: code = NotFound desc = could not find container \"8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a\": container with ID starting with 8dee2fd2d0c1bee969e0576608c8589d242287b83ddf63b1d3e12204ba1f611a not found: ID does not exist"
Oct 31 15:11:21.179643 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.179560    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:11:21.183397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.183365    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-gh2k6"]
Oct 31 15:11:21.420557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:21.420523    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" path="/var/lib/kubelet/pods/dad18e79-ee73-4f4c-bbfb-8c801d0613bd/volumes"
Oct 31 15:11:37.019389 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:11:37.019346    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:11:37.067617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.067537    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:11:37.068270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.068247    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" containerName="maas-api"
Oct 31 15:11:37.068270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.068269    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" containerName="maas-api"
Oct 31 15:11:37.068467 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.068439    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad18e79-ee73-4f4c-bbfb-8c801d0613bd" containerName="maas-api"
Oct 31 15:11:37.073461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.073438    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.100758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.100724    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.100939 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.100769    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbchx\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.101924 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.101895    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.101997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.101938    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.102056 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.102020    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.102199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.102183    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.102279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.102216    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.102362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.102339    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.102443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.102426    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.125347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.125298    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-644a8a-openshift-default-dockercfg-5qxv9\""
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204540    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204662    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204692    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204739    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204812    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204889    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.204929    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbchx\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.205057    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.205087    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.206145    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.206249    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.206675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.215423    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.217657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.215638    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.231904 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.231841    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.238542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.238506    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.286170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.286039    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:11:37.286508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.286486    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.292266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.292232    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbchx\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.392081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.392047    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:37.837709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.837670    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:11:37.874042 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:11:37.874004    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f0e3dc_23de_44ca_a40b_c0e9d3d2f58d.slice/crio-3492de0626ac9ed06c48104d4df4f91e4363015da87607324d21765fcb3f64d7 WatchSource:0}: Error finding container 3492de0626ac9ed06c48104d4df4f91e4363015da87607324d21765fcb3f64d7: Status 404 returned error can't find the container with id 3492de0626ac9ed06c48104d4df4f91e4363015da87607324d21765fcb3f64d7
Oct 31 15:11:37.876530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.876497    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:37.876639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.876585    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:37.876639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:37.876622    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:38.182173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.182092    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" event={"ID":"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d","Type":"ContainerStarted","Data":"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"}
Oct 31 15:11:38.182173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.182150    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" event={"ID":"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d","Type":"ContainerStarted","Data":"3492de0626ac9ed06c48104d4df4f91e4363015da87607324d21765fcb3f64d7"}
Oct 31 15:11:38.302514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.302443    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podStartSLOduration=1.302423456 podStartE2EDuration="1.302423456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:38.296910986 +0000 UTC m=+1539.023192001" watchObservedRunningTime="2026-10-31 15:11:38.302423456 +0000 UTC m=+1539.028704435"
Oct 31 15:11:38.392855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.392821    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:38.394134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.394107    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:38.394276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:38.394159    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:39.393215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:39.393173    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:39.393657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:39.393246    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:39.836941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:39.835635    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": read tcp 10.134.0.2:52472->10.134.0.92:15021: read: connection reset by peer" start-of-body=
Oct 31 15:11:39.836941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:39.835701    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": read tcp 10.134.0.2:52472->10.134.0.92:15021: read: connection reset by peer"
Oct 31 15:11:40.227733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.227640    4549 generic.go:356] "Generic (PLEG): container finished" podID="4635b899-8973-4181-afa5-c45aa5471013" containerID="ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8" exitCode=0
Oct 31 15:11:40.227883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.227719    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerDied","Data":"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"}
Oct 31 15:11:40.228288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.228261    4549 scope.go:122] "RemoveContainer" containerID="ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"
Oct 31 15:11:40.229842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.229810    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:40.229950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.229893    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:40.229950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.229944    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:11:40.393105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.393069    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:40.393263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:40.393130    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:41.273406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.273370    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerStarted","Data":"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"}
Oct 31 15:11:41.392913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.392881    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:41.393073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.392936    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:41.862738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.862685    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:11:41.863286 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.862983    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy" containerID="cri-o://5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206" gracePeriod=30
Oct 31 15:11:41.987889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.982644    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:11:41.987889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.982768    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 15:11:41.987889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:41.982820    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 15:11:42.392514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:42.392425    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:42.392756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:42.392515    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:42.974432 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:42.974402    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 15:11:42.974961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:42.974935    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 15:11:43.393098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:43.393061    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 15:11:43.393296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:43.393138    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 15:11:43.974788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:43.974756    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 15:11:43.975184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:43.974819    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 15:11:44.537601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:44.536732    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:44.538250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:44.538226    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:44.540087 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:44.539431    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:11:44.974739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:44.974453    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 15:11:44.975031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:44.975002    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 15:11:45.978234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:45.978149    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 15:11:45.978234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:45.978216    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 15:11:46.984937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:46.984907    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:11:46.985962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:46.985922    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:11:46.987241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:46.987213    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:11:47.176890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.176863    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:11:47.263917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.263887    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.263923    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.263959    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264022    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264045    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264134    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264192    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264191    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:11:47.264344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264243    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264316    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:11:47.264640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264345    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\") pod \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\" (UID: \"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2\") "
Oct 31 15:11:47.264640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264510    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:11:47.265037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264788    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:11:47.265037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.264860    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:11:47.265397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.265374    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.265464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.265405    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.265464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.265421    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.265464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.265437    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.265464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.265452    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.266864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.266826    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:11:47.266969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.266868    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:11:47.266969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.266892    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:11:47.267076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.267052    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f" pod "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" (UID: "eb1ab48f-4d19-41bb-a498-5b3bca24c1f2"). InnerVolumeSpecName "kube-api-access-76b5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:11:47.366982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.366942    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.366982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.366985    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.367211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.367000    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.367211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.367014    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76b5f\" (UniqueName: \"kubernetes.io/projected/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2-kube-api-access-76b5f\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:11:47.377067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.377034    4549 generic.go:356] "Generic (PLEG): container finished" podID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerID="5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206" exitCode=0
Oct 31 15:11:47.377212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.377087    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"
Oct 31 15:11:47.377212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.377113    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" event={"ID":"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2","Type":"ContainerDied","Data":"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"}
Oct 31 15:11:47.377212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.377172    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h" event={"ID":"eb1ab48f-4d19-41bb-a498-5b3bca24c1f2","Type":"ContainerDied","Data":"1628bd5f9c345c1eca73b406862cef35807630884c7f3178d83b7b9750ae34b3"}
Oct 31 15:11:47.377212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.377199    4549 scope.go:122] "RemoveContainer" containerID="5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"
Oct 31 15:11:47.388717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.388696    4549 scope.go:122] "RemoveContainer" containerID="5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"
Oct 31 15:11:47.389000 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:11:47.388980    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206\": container with ID starting with 5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206 not found: ID does not exist" containerID="5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"
Oct 31 15:11:47.389068 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.389015    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206"} err="failed to get container status \"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206\": rpc error: code = NotFound desc = could not find container \"5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206\": container with ID starting with 5d7bfeffc1b96bff8547915b372f7b775054ad026abab344754918e6811f1206 not found: ID does not exist"
Oct 31 15:11:47.520118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.519633    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:11:47.527245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:47.527216    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-zcc7h"]
Oct 31 15:11:49.423883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:11:49.423842    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" path="/var/lib/kubelet/pods/eb1ab48f-4d19-41bb-a498-5b3bca24c1f2/volumes"
Oct 31 15:12:07.019769 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:12:07.019723    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:12:22.515949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.515876    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:12:22.516775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.516751    4549 cpu_manager.go:414] "containerMap: removing container" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy"
Oct 31 15:12:22.516775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.516775    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy"
Oct 31 15:12:22.516998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.516925    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1ab48f-4d19-41bb-a498-5b3bca24c1f2" containerName="istio-proxy"
Oct 31 15:12:22.520444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.520426    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.565035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.565009    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:12:22.569033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.569018    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.614460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614425    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.614659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614484    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.614659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614560    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.614797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614661    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.614797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614702    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.614797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614770    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.614797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614793    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614810    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.615038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614855    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.615038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614877    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.615038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.614904    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615060    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615086    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615126    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615151    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615177    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5qzh\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615223    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ln2c\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.615240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.615242    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.618612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.618537    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:12:22.629231 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.629208    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:12:22.718217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718186    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718224    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718287    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.718424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718338    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718378    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5qzh\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718459    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ln2c\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718491    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718537    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718608    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718640    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.718723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718681    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718734    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718807    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718841    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718863    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718930    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718955    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.719029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.718984    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.719383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.719237    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.719443    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.719735    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.720155    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.720227    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.720840    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.721064    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.721280    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.721591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.721520    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.722406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.722362    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.724368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.724348    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.724505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.724407    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.724653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.724634    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.725143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.725123    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.775941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.775871    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.776085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.775958    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.776085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.775979    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ln2c\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.776085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.776037    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5qzh\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:22.838356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.838324    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:22.884471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:22.884438    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:23.271917 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:12:23.271883    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a77e321_f28b_4355_9187_b78c51baca4e.slice/crio-cef70f0504a8effcad5400c496aa353b7b6757a98d77ebcd69a76a9ac91468e5 WatchSource:0}: Error finding container cef70f0504a8effcad5400c496aa353b7b6757a98d77ebcd69a76a9ac91468e5: Status 404 returned error can't find the container with id cef70f0504a8effcad5400c496aa353b7b6757a98d77ebcd69a76a9ac91468e5
Oct 31 15:12:23.274294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.274267    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:12:23.274555 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.274525    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.274677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.274639    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.274677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.274673    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.386509 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:12:23.386469    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf72c8fc_3726_4a90_b5a2_c0681ea96e7c.slice/crio-e668262faac28256fe66dc4c6d4e34e8d4e18928835c0d820368df4f5c355602 WatchSource:0}: Error finding container e668262faac28256fe66dc4c6d4e34e8d4e18928835c0d820368df4f5c355602: Status 404 returned error can't find the container with id e668262faac28256fe66dc4c6d4e34e8d4e18928835c0d820368df4f5c355602
Oct 31 15:12:23.387841 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.387812    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:12:23.390098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.390062    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.390213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.390129    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.390213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.390157    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:12:23.728167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.728072    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:12:23.728677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.728338    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" containerID="cri-o://e8f443ba3be8cd1f54c48cb2d15cfb020facfe5fb3cc12703553ee43055ae361" gracePeriod=30
Oct 31 15:12:23.777377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.777335    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" event={"ID":"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c","Type":"ContainerStarted","Data":"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"}
Oct 31 15:12:23.777622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.777404    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" event={"ID":"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c","Type":"ContainerStarted","Data":"e668262faac28256fe66dc4c6d4e34e8d4e18928835c0d820368df4f5c355602"}
Oct 31 15:12:23.786164 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.786132    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" event={"ID":"2a77e321-f28b-4355-9187-b78c51baca4e","Type":"ContainerStarted","Data":"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"}
Oct 31 15:12:23.786493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.786381    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" event={"ID":"2a77e321-f28b-4355-9187-b78c51baca4e","Type":"ContainerStarted","Data":"cef70f0504a8effcad5400c496aa353b7b6757a98d77ebcd69a76a9ac91468e5"}
Oct 31 15:12:23.840103 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.838447    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:23.885019 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.884988    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:23.990652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:23.986373    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podStartSLOduration=1.98635446 podStartE2EDuration="1.98635446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:23.982381884 +0000 UTC m=+1584.708662872" watchObservedRunningTime="2026-10-31 15:12:23.98635446 +0000 UTC m=+1584.712635440"
Oct 31 15:12:24.117593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:24.116508    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podStartSLOduration=2.116485458 podStartE2EDuration="2.116485458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:24.113118615 +0000 UTC m=+1584.839399605" watchObservedRunningTime="2026-10-31 15:12:24.116485458 +0000 UTC m=+1584.842766437"
Oct 31 15:12:24.849602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:24.848698    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 15:12:24.849602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:24.848757    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 15:12:24.887360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:24.887326    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:24.887559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:24.887412    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:25.843634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:25.843601    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 15:12:25.843938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:25.843911    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 15:12:25.885614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:25.885543    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:25.886206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:25.886164    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:26.848595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:26.848485    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 15:12:26.848595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:26.848533    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 15:12:26.884998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:26.884970    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:26.885169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:26.885017    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:27.839594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:27.838721    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 15:12:27.839594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:27.838781    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 15:12:27.884852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:27.884823    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:27.885023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:27.884871    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:28.842609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.840667    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 15:12:28.842609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.840744    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 15:12:28.881538 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.881507    4549 generic.go:356] "Generic (PLEG): container finished" podID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerID="e8f443ba3be8cd1f54c48cb2d15cfb020facfe5fb3cc12703553ee43055ae361" exitCode=0
Oct 31 15:12:28.881710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.881603    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" event={"ID":"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf","Type":"ContainerDied","Data":"e8f443ba3be8cd1f54c48cb2d15cfb020facfe5fb3cc12703553ee43055ae361"}
Oct 31 15:12:28.885333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.885308    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:28.885427 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.885373    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:28.954595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.954436    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 15:12:28.954595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:28.954511    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 15:12:29.216966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.216933    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:12:29.294893 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.294857    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.294919    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.294951    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.294995    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295049    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295137    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295252    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295319    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295443    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:12:29.295506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295447    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\") pod \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\" (UID: \"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf\") "
Oct 31 15:12:29.295944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.295926    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:12:29.296098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.296081    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.296242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.296227    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:12:29.296722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.296651    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:12:29.296853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.296812    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:12:29.322603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.319705    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:12:29.322603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.321064    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:12:29.334598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.331801    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "kube-api-access-qmrfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:12:29.355869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.355813    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy" pod "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" (UID: "32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398112    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398151    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrfs\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-kube-api-access-qmrfs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398167    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398183    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398198    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398212    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398225    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.410598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.398238    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:29.885325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.885284    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 15:12:29.885918 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.885344    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 15:12:29.892185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.892159    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:29.892398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.892371    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc" event={"ID":"32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf","Type":"ContainerDied","Data":"4d7df347a9f04a88f859e8288e57bd1f1eb342536f4996e88e83d48512b1fccf"}
Oct 31 15:12:29.892674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.892421    4549 scope.go:122] "RemoveContainer" containerID="e8f443ba3be8cd1f54c48cb2d15cfb020facfe5fb3cc12703553ee43055ae361"
Oct 31 15:12:29.892674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.892533    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"
Oct 31 15:12:29.894308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.894275    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:29.897292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:29.897270    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:12:30.120553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:30.120515    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:12:30.124236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:30.124211    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-cb2sc"]
Oct 31 15:12:30.890712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:30.890684    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:30.891116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:30.891100    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:30.891922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:30.891904    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:12:31.421734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:31.421697    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" path="/var/lib/kubelet/pods/32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf/volumes"
Oct 31 15:12:37.013762 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:12:37.013709    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:12:40.238273 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.238240    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:12:40.238685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.238448    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" podUID="0212fd24-d483-41cb-affa-c144195698c6" containerName="maas-api" containerID="cri-o://05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167" gracePeriod=30
Oct 31 15:12:40.905619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.905590    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:12:40.980390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.980355    4549 generic.go:356] "Generic (PLEG): container finished" podID="0212fd24-d483-41cb-affa-c144195698c6" containerID="05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167" exitCode=0
Oct 31 15:12:40.980597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.980399    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"
Oct 31 15:12:40.980597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.980448    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" event={"ID":"0212fd24-d483-41cb-affa-c144195698c6","Type":"ContainerDied","Data":"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"}
Oct 31 15:12:40.980597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.980533    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2" event={"ID":"0212fd24-d483-41cb-affa-c144195698c6","Type":"ContainerDied","Data":"33976ab76bad88c53fb44428497a00e29f7f9c981dec401ddf078b2432bfd0dd"}
Oct 31 15:12:40.980597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.980559    4549 scope.go:122] "RemoveContainer" containerID="05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"
Oct 31 15:12:40.991006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.990990    4549 scope.go:122] "RemoveContainer" containerID="05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"
Oct 31 15:12:40.991295 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:12:40.991275    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167\": container with ID starting with 05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167 not found: ID does not exist" containerID="05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"
Oct 31 15:12:40.991349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.991305    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167"} err="failed to get container status \"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167\": rpc error: code = NotFound desc = could not find container \"05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167\": container with ID starting with 05a6216425be073d2149049d6594b9133e8aec2b6d25dbdcd24abedbe97a1167 not found: ID does not exist"
Oct 31 15:12:40.995906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.995883    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") pod \"0212fd24-d483-41cb-affa-c144195698c6\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") "
Oct 31 15:12:40.995996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.995962    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\") pod \"0212fd24-d483-41cb-affa-c144195698c6\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") "
Oct 31 15:12:40.996092 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.996078    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\") pod \"0212fd24-d483-41cb-affa-c144195698c6\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") "
Oct 31 15:12:40.996251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.996229    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\" (UniqueName: \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\") pod \"0212fd24-d483-41cb-affa-c144195698c6\" (UID: \"0212fd24-d483-41cb-affa-c144195698c6\") "
Oct 31 15:12:40.996367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.996340    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca" pod "0212fd24-d483-41cb-affa-c144195698c6" (UID: "0212fd24-d483-41cb-affa-c144195698c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:12:40.996773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.996715    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0212fd24-d483-41cb-affa-c144195698c6-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:40.998214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.998194    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls" pod "0212fd24-d483-41cb-affa-c144195698c6" (UID: "0212fd24-d483-41cb-affa-c144195698c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:12:40.998402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.998372    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs" pod "0212fd24-d483-41cb-affa-c144195698c6" (UID: "0212fd24-d483-41cb-affa-c144195698c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:12:40.998402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:40.998368    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl" pod "0212fd24-d483-41cb-affa-c144195698c6" (UID: "0212fd24-d483-41cb-affa-c144195698c6"). InnerVolumeSpecName "kube-api-access-fdnwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:12:41.098197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:41.098118    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:41.098197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:41.098145    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0212fd24-d483-41cb-affa-c144195698c6-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:41.098197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:41.098156    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdnwl\" (UniqueName: \"kubernetes.io/projected/0212fd24-d483-41cb-affa-c144195698c6-kube-api-access-fdnwl\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:41.433898 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:41.433820    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:12:41.483267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:41.483234    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4d3ea8-67d5449b6f-x4rp2"]
Oct 31 15:12:43.420787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:43.420749    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0212fd24-d483-41cb-affa-c144195698c6" path="/var/lib/kubelet/pods/0212fd24-d483-41cb-affa-c144195698c6/volumes"
Oct 31 15:12:51.806591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:51.805503    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:12:51.806591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:51.805944    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" podUID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" containerName="maas-api" containerID="cri-o://67c5bd219bc57d98a73b0e438f6a0eb7aa2002fadcb376caaf62f456e8d00a88" gracePeriod=30
Oct 31 15:12:52.310597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.307937    4549 generic.go:356] "Generic (PLEG): container finished" podID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" containerID="67c5bd219bc57d98a73b0e438f6a0eb7aa2002fadcb376caaf62f456e8d00a88" exitCode=0
Oct 31 15:12:52.310597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.308198    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" event={"ID":"929a3d31-fff6-4e55-bdcb-7e04566d6dca","Type":"ContainerDied","Data":"67c5bd219bc57d98a73b0e438f6a0eb7aa2002fadcb376caaf62f456e8d00a88"}
Oct 31 15:12:52.543291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.543258    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:12:52.659333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.659244    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\") pod \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") "
Oct 31 15:12:52.659512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.659407    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") pod \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") "
Oct 31 15:12:52.659636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.659517    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\") pod \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") "
Oct 31 15:12:52.659636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.659587    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\" (UniqueName: \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\") pod \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\" (UID: \"929a3d31-fff6-4e55-bdcb-7e04566d6dca\") "
Oct 31 15:12:52.660180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.660148    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca" pod "929a3d31-fff6-4e55-bdcb-7e04566d6dca" (UID: "929a3d31-fff6-4e55-bdcb-7e04566d6dca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:12:52.660289 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.660272    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/929a3d31-fff6-4e55-bdcb-7e04566d6dca-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:52.662636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.662603    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs" pod "929a3d31-fff6-4e55-bdcb-7e04566d6dca" (UID: "929a3d31-fff6-4e55-bdcb-7e04566d6dca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:12:52.662750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.662724    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls" pod "929a3d31-fff6-4e55-bdcb-7e04566d6dca" (UID: "929a3d31-fff6-4e55-bdcb-7e04566d6dca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:12:52.664197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.664170    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj" pod "929a3d31-fff6-4e55-bdcb-7e04566d6dca" (UID: "929a3d31-fff6-4e55-bdcb-7e04566d6dca"). InnerVolumeSpecName "kube-api-access-qqsdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:12:52.761585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.761551    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:52.761585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.761591    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqsdj\" (UniqueName: \"kubernetes.io/projected/929a3d31-fff6-4e55-bdcb-7e04566d6dca-kube-api-access-qqsdj\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:52.761787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:52.761602    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/929a3d31-fff6-4e55-bdcb-7e04566d6dca-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:12:53.318474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:53.318426    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg" event={"ID":"929a3d31-fff6-4e55-bdcb-7e04566d6dca","Type":"ContainerDied","Data":"3f7c1eb3e48ddd97d2187ad563f4fcc94a117496dd2e04ff2ec46a3cab5e2dcd"}
Oct 31 15:12:53.319028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:53.318558    4549 scope.go:122] "RemoveContainer" containerID="67c5bd219bc57d98a73b0e438f6a0eb7aa2002fadcb376caaf62f456e8d00a88"
Oct 31 15:12:53.319028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:53.318767    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"
Oct 31 15:12:53.484948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:53.484914    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:12:53.533614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:53.533562    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5f17a1-7886df7fdb-4zlqg"]
Oct 31 15:12:55.424023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:12:55.423987    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" path="/var/lib/kubelet/pods/929a3d31-fff6-4e55-bdcb-7e04566d6dca/volumes"
Oct 31 15:13:04.646402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.646356    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:13:04.647243 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647225    4549 cpu_manager.go:414] "containerMap: removing container" podUID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" containerName="maas-api"
Oct 31 15:13:04.647291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647246    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" containerName="maas-api"
Oct 31 15:13:04.647291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647276    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0212fd24-d483-41cb-affa-c144195698c6" containerName="maas-api"
Oct 31 15:13:04.647291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647284    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0212fd24-d483-41cb-affa-c144195698c6" containerName="maas-api"
Oct 31 15:13:04.647383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647328    4549 cpu_manager.go:414] "containerMap: removing container" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy"
Oct 31 15:13:04.647383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647336    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy"
Oct 31 15:13:04.647479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647468    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="929a3d31-fff6-4e55-bdcb-7e04566d6dca" containerName="maas-api"
Oct 31 15:13:04.647511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647488    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ab3497-76d8-4b28-b9fb-8bd80ec5f4bf" containerName="istio-proxy"
Oct 31 15:13:04.647511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.647499    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0212fd24-d483-41cb-affa-c144195698c6" containerName="maas-api"
Oct 31 15:13:04.653071 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.653052    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.706097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.706065    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:13:04.749986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.749954    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5695787b-openshift-default-dockercfg-zpbf4\""
Oct 31 15:13:04.763456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763425    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763493    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763590    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763726    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763807    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxw84\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763935 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763922    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763947    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.763981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763965    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.764050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.763991    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865202    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865273    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865301    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865376    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865448    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxw84\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865524    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865554    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865619    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865660    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865697    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865894 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865840    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.865950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.865891    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.866092 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.866071    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.866319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.866297    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.867848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.867825    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.868253 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.868238    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.922539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.922463    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.922739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.922717    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxw84\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-rz94v\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:04.966825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:04.966784    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:05.453184 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:13:05.453155    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4074d5_50df_4721_9fc5_9460494d843f.slice/crio-bc2eda3b4ce971516d71aee86fc0d304000a09308dedbc3e7e8fa09711f0c83f WatchSource:0}: Error finding container bc2eda3b4ce971516d71aee86fc0d304000a09308dedbc3e7e8fa09711f0c83f: Status 404 returned error can't find the container with id bc2eda3b4ce971516d71aee86fc0d304000a09308dedbc3e7e8fa09711f0c83f
Oct 31 15:13:05.454108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:05.454010    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:13:05.457611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:05.457283    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:05.457611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:05.457451    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:05.457611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:05.457548    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:06.440244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.440204    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" event={"ID":"0d4074d5-50df-4721-9fc5-9460494d843f","Type":"ContainerStarted","Data":"be40f651f0280704ff9965ed364ef0a2b426e2f7aa4e433744859ec3044bb9ff"}
Oct 31 15:13:06.440794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.440268    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" event={"ID":"0d4074d5-50df-4721-9fc5-9460494d843f","Type":"ContainerStarted","Data":"bc2eda3b4ce971516d71aee86fc0d304000a09308dedbc3e7e8fa09711f0c83f"}
Oct 31 15:13:06.615203 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.615139    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podStartSLOduration=2.615119593 podStartE2EDuration="2.615119593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:06.608365198 +0000 UTC m=+1627.334646176" watchObservedRunningTime="2026-10-31 15:13:06.615119593 +0000 UTC m=+1627.341400573"
Oct 31 15:13:06.967991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.967957    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:06.969407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.969389    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:06.969476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:06.969444    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:07.043251 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:13:07.043210    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:13:07.967128 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:07.967095    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:07.967529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:07.967146    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:08.967684 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:08.967640    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:08.968093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:08.967713    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:09.968144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:09.968102    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:09.968680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:09.968176    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:10.967651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:10.967613    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:10.967821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:10.967678    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:11.969729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:11.969687    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 15:13:11.970225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:11.969754    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 15:13:12.979421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:12.979379    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:12.980197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:12.980167    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:12.981423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:12.981401    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:13:19.775551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.775510    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:13:19.779690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.779666    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927285    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927435    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927479    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927500    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927684    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927716    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927814    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxhv\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927833    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:19.927864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:19.927859    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.010874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.010829    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:13:20.028907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.028820    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.028907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.028887    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.028980    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxhv\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029003    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029038    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029081    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029218    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029265    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029284    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.029332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029299    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.030002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.029959    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.030321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.030299    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.030474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.030452    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.030808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.030789    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.032357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.032331    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.033438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.033413    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.086532 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.086500    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.086687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.086675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxhv\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-g74hh\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.093676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.093605    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:20.520136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.520104    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:13:20.560640 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:13:20.560607    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da3adf0_55ca_4426_b398_fd0116716784.slice/crio-74e85f6e6da3c1386791822d43af9b0a727ca98ecd626d54eed8205afcbdaa97 WatchSource:0}: Error finding container 74e85f6e6da3c1386791822d43af9b0a727ca98ecd626d54eed8205afcbdaa97: Status 404 returned error can't find the container with id 74e85f6e6da3c1386791822d43af9b0a727ca98ecd626d54eed8205afcbdaa97
Oct 31 15:13:20.563246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.563212    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:20.563321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.563284    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:20.563321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:20.563315    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:21.571773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:21.571733    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" event={"ID":"2da3adf0-55ca-4426-b398-fd0116716784","Type":"ContainerStarted","Data":"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"}
Oct 31 15:13:21.572163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:21.571799    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" event={"ID":"2da3adf0-55ca-4426-b398-fd0116716784","Type":"ContainerStarted","Data":"74e85f6e6da3c1386791822d43af9b0a727ca98ecd626d54eed8205afcbdaa97"}
Oct 31 15:13:21.737334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:21.737266    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podStartSLOduration=2.737246912 podStartE2EDuration="2.737246912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:21.736979814 +0000 UTC m=+1642.463260799" watchObservedRunningTime="2026-10-31 15:13:21.737246912 +0000 UTC m=+1642.463527891"
Oct 31 15:13:22.094386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:22.094352    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:22.096173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:22.096147    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:22.096303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:22.096199    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:23.094623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:23.094562    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:23.095009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:23.094643    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:24.093986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.093952    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:24.094175 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.094013    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:24.783861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.783831    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7585-b5fdb8"]
Oct 31 15:13:24.788116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.788099    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:24.844874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.844840    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-b5fdb8"]
Oct 31 15:13:24.887226 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.887187    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\""
Oct 31 15:13:24.908710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:24.908675    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnglh\" (UniqueName: \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\") pod \"test-curl-7585-b5fdb8\" (UID: \"7f09647d-f8a9-470c-8795-dd47837bd503\") " pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:25.011863 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.011822    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnglh\" (UniqueName: \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\") pod \"test-curl-7585-b5fdb8\" (UID: \"7f09647d-f8a9-470c-8795-dd47837bd503\") " pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:25.069618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.069448    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnglh\" (UniqueName: \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\") pod \"test-curl-7585-b5fdb8\" (UID: \"7f09647d-f8a9-470c-8795-dd47837bd503\") " pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:25.094517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.094487    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:25.094708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.094550    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:25.119412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.118203    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:25.596312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:25.596283    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7585-b5fdb8"]
Oct 31 15:13:26.094395 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:26.094357    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:26.094809 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:26.094422    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:26.637885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:26.637848    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-b5fdb8" event={"ID":"7f09647d-f8a9-470c-8795-dd47837bd503","Type":"ContainerStarted","Data":"5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b"}
Oct 31 15:13:26.638052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:26.637902    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7585-b5fdb8" event={"ID":"7f09647d-f8a9-470c-8795-dd47837bd503","Type":"ContainerStarted","Data":"ad67d8787e446238cb6f24a6af4b074bebc7216dfc5d5bd056756a0253c3c8b9"}
Oct 31 15:13:26.748895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:26.748842    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7585-b5fdb8" podStartSLOduration=2.7488260860000002 podStartE2EDuration="2.748826086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:26.74765989 +0000 UTC m=+1647.473940866" watchObservedRunningTime="2026-10-31 15:13:26.748826086 +0000 UTC m=+1647.475107103"
Oct 31 15:13:27.094522 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:27.094481    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 15:13:27.095057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:27.094550    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 15:13:28.100563 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.100523    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:28.113867 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.101900    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:28.113867 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.103130    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:13:28.235140 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.230664    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7585-b5fdb8"]
Oct 31 15:13:28.235140 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.232014    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7585-b5fdb8"]
Oct 31 15:13:28.699991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:28.697080    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7585-b5fdb8" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" containerName="test-curl-7585-b5fdb8" containerID="cri-o://5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b" gracePeriod=2
Oct 31 15:13:28.750625 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:13:28.747975    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b5fdb8\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:29.469810 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:13:29.469757    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7585-b5fdb8\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:30.969963 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:30.969937    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:31.082471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.082440    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\" (UniqueName: \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\") pod \"7f09647d-f8a9-470c-8795-dd47837bd503\" (UID: \"7f09647d-f8a9-470c-8795-dd47837bd503\") "
Oct 31 15:13:31.084780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.084750    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh" pod "7f09647d-f8a9-470c-8795-dd47837bd503" (UID: "7f09647d-f8a9-470c-8795-dd47837bd503"). InnerVolumeSpecName "kube-api-access-vnglh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:13:31.184626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.184557    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnglh\" (UniqueName: \"kubernetes.io/projected/7f09647d-f8a9-470c-8795-dd47837bd503-kube-api-access-vnglh\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:31.423276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.423180    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" path="/var/lib/kubelet/pods/7f09647d-f8a9-470c-8795-dd47837bd503/volumes"
Oct 31 15:13:31.712446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.712358    4549 generic.go:356] "Generic (PLEG): container finished" podID="7f09647d-f8a9-470c-8795-dd47837bd503" containerID="5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b" exitCode=137
Oct 31 15:13:31.712446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.712411    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7585-b5fdb8"
Oct 31 15:13:31.712446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.712426    4549 scope.go:122] "RemoveContainer" containerID="5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b"
Oct 31 15:13:31.723253 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.723047    4549 scope.go:122] "RemoveContainer" containerID="5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b"
Oct 31 15:13:31.723366 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:13:31.723347    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b\": container with ID starting with 5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b not found: ID does not exist" containerID="5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b"
Oct 31 15:13:31.723405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:31.723377    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b"} err="failed to get container status \"5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b\": rpc error: code = NotFound desc = could not find container \"5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b\": container with ID starting with 5f137f2606c03d0498e09723f4580d8662b5a1f09229d02e36a7f9f00825758b not found: ID does not exist"
Oct 31 15:13:33.652238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:33.652197    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:13:34.901356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:34.901316    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:13:34.902181 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:34.902158    4549 cpu_manager.go:414] "containerMap: removing container" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" containerName="test-curl-7585-b5fdb8"
Oct 31 15:13:34.902289 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:34.902184    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" containerName="test-curl-7585-b5fdb8"
Oct 31 15:13:34.902357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:34.902342    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f09647d-f8a9-470c-8795-dd47837bd503" containerName="test-curl-7585-b5fdb8"
Oct 31 15:13:34.907348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:34.907326    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.038869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.038829    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.038940    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.038983    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039029    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039113    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039172    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039255    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039278    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q7xh\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.039359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.039313    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.103444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.103406    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:13:35.140959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.140918    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141007    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141095    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141134    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q7xh\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141168    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141316    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141398    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141424    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141435    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141665 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141506    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.141890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.141861    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.142041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.142007    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.142158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.142044    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.142646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.142622    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.144874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.144847    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.145394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.145373    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.198496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.198421    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.198672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.198656    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q7xh\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-48vdq\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.225622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.221888    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:35.659193 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:13:35.659168    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741386e2_83f2_4eb8_95cf_dd56528a944d.slice/crio-9d53ec12c84c9a6dc04f40af404b11a6a933312e452ded1fd108732dbef53dfa WatchSource:0}: Error finding container 9d53ec12c84c9a6dc04f40af404b11a6a933312e452ded1fd108732dbef53dfa: Status 404 returned error can't find the container with id 9d53ec12c84c9a6dc04f40af404b11a6a933312e452ded1fd108732dbef53dfa
Oct 31 15:13:35.659354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.659334    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:13:35.661855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.661825    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:35.661899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.661889    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:35.661934 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.661917    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:13:35.744151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.744119    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" event={"ID":"741386e2-83f2-4eb8-95cf-dd56528a944d","Type":"ContainerStarted","Data":"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"}
Oct 31 15:13:35.744356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.744181    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" event={"ID":"741386e2-83f2-4eb8-95cf-dd56528a944d","Type":"ContainerStarted","Data":"9d53ec12c84c9a6dc04f40af404b11a6a933312e452ded1fd108732dbef53dfa"}
Oct 31 15:13:35.864003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:35.863953    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podStartSLOduration=1.86393796 podStartE2EDuration="1.86393796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:35.858767381 +0000 UTC m=+1656.585048395" watchObservedRunningTime="2026-10-31 15:13:35.86393796 +0000 UTC m=+1656.590218939"
Oct 31 15:13:36.225593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:36.224915    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:36.230589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:36.230440    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:36.230589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:36.230503    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:37.037599 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:13:37.037548    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:13:37.226637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:37.223885    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:37.226637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:37.223946    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:38.223324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:38.223181    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:38.223324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:38.223243    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:39.223598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:39.223033    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:39.223598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:39.223096    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:40.225595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:40.224912    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:40.225595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:40.224971    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:41.227015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:41.226974    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Oct 31 15:13:41.227687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:41.227657    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Oct 31 15:13:42.271381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:42.270923    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:42.271954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:42.271889    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:42.272686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:42.272666    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:13:48.676881 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:48.676829    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main" containerID="cri-o://48daf72589938ce1742197833bc920d0bc7804b68781159d9166a5f06bf801fb" gracePeriod=45
Oct 31 15:13:48.859928 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:48.859897    4549 generic.go:356] "Generic (PLEG): container finished" podID="469f9541-1893-4d0a-899a-698108d2c6e4" containerID="48daf72589938ce1742197833bc920d0bc7804b68781159d9166a5f06bf801fb" exitCode=0
Oct 31 15:13:48.860098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:48.860005    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" event={"ID":"469f9541-1893-4d0a-899a-698108d2c6e4","Type":"ContainerDied","Data":"48daf72589938ce1742197833bc920d0bc7804b68781159d9166a5f06bf801fb"}
Oct 31 15:13:48.951033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:48.951009    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:13:49.003816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.003781    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004012 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.003865    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004012 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.003938    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004011    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\" (UniqueName: \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004088    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004141    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:13:49.004219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004186    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\") pod \"469f9541-1893-4d0a-899a-698108d2c6e4\" (UID: \"469f9541-1893-4d0a-899a-698108d2c6e4\") "
Oct 31 15:13:49.004380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004337    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:13:49.004492 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004469    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:13:49.004941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004924    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.004994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004948    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.004994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.004962    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.006369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.006343    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:13:49.006734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.006716    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:13:49.006799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.006777    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98" pod "469f9541-1893-4d0a-899a-698108d2c6e4" (UID: "469f9541-1893-4d0a-899a-698108d2c6e4"). InnerVolumeSpecName "kube-api-access-vjp98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:13:49.107184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.107145    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/469f9541-1893-4d0a-899a-698108d2c6e4-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.107184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.107189    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/469f9541-1893-4d0a-899a-698108d2c6e4-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.107388 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.107206    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjp98\" (UniqueName: \"kubernetes.io/projected/469f9541-1893-4d0a-899a-698108d2c6e4-kube-api-access-vjp98\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:13:49.870359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.870328    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"
Oct 31 15:13:49.870788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.870356    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28" event={"ID":"469f9541-1893-4d0a-899a-698108d2c6e4","Type":"ContainerDied","Data":"4e617528fea7ca53b8e66956a7bec9cee84608f6065cbb8c03d7764cd83beccc"}
Oct 31 15:13:49.870788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:49.870411    4549 scope.go:122] "RemoveContainer" containerID="48daf72589938ce1742197833bc920d0bc7804b68781159d9166a5f06bf801fb"
Oct 31 15:13:50.034592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:50.034548    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:13:50.038224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:50.038201    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5695787b/e2e-lifecycle-model-5695787b-kserve-5dcb556cc-6kh28"]
Oct 31 15:13:51.422666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:51.422629    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" path="/var/lib/kubelet/pods/469f9541-1893-4d0a-899a-698108d2c6e4/volumes"
Oct 31 15:13:58.303410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.303374    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:13:58.303952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.303674    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" containerID="cri-o://5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7" gracePeriod=30
Oct 31 15:13:58.313709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.313678    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:13:58.313968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.313924    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" containerID="cri-o://de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f" gracePeriod=30
Oct 31 15:13:58.333646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.333615    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:13:58.333882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:58.333857    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" containerID="cri-o://d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a" gracePeriod=30
Oct 31 15:13:59.895215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:59.895178    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:13:59.895660 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:13:59.895245    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:00.893809 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:00.893769    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:00.894107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:00.894081    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:01.988009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:01.987974    4549 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:01.988454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:01.988044    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:03.638791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.638760    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:14:03.666675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.666646    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:14:03.687085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.687065    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:14:03.752478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752451    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.752674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752487    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.752674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752510    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.752674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752554    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752608    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.752674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752650    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752705    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752733    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752786    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752829    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752916    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.752957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752949    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.752983    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753016    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753055    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753131    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753142    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753150    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753188    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.753407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753184    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753439    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753500    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753510    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753646    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753726    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753798    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.753837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753833    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.754237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753960    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.753990    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754194    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754372    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754593    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754598    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.754739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754680    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.754739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754721    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.754860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754756    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.754860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754787    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\") pod \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\" (UID: \"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c\") "
Oct 31 15:14:03.754860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754816    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\") pod \"2a77e321-f28b-4355-9187-b78c51baca4e\" (UID: \"2a77e321-f28b-4355-9187-b78c51baca4e\") "
Oct 31 15:14:03.755023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.754867    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\") pod \"4635b899-8973-4181-afa5-c45aa5471013\" (UID: \"4635b899-8973-4181-afa5-c45aa5471013\") "
Oct 31 15:14:03.755229 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755202    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.755559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755527    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:03.755855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755827    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755852    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a77e321-f28b-4355-9187-b78c51baca4e-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755875    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a77e321-f28b-4355-9187-b78c51baca4e-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755888    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755904    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755904    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755916    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4635b899-8973-4181-afa5-c45aa5471013-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755929    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.755948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755942    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755954    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755967    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755980    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.755993    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.756005    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.756018    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.756031    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.756408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.756107    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.758450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.758425    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:03.758648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.758501    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "kube-api-access-cb2hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.758648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.758561    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.758829 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.758807    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.758930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.758909    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "kube-api-access-v5qzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.759057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.759033    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token" pod "4635b899-8973-4181-afa5-c45aa5471013" (UID: "4635b899-8973-4181-afa5-c45aa5471013"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.759545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.759516    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.759711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.759552    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "kube-api-access-2ln2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:03.760404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.760382    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy" pod "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" (UID: "cf72c8fc-3726-4a90-b5a2-c0681ea96e7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.760404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.760389    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy" pod "2a77e321-f28b-4355-9187-b78c51baca4e" (UID: "2a77e321-f28b-4355-9187-b78c51baca4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:03.857435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857361    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857398    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2hz\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-kube-api-access-cb2hz\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857415    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5qzh\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-kube-api-access-v5qzh\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857429    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857443    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857457    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a77e321-f28b-4355-9187-b78c51baca4e-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857471    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4635b899-8973-4181-afa5-c45aa5471013-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857484    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857497    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857511    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4635b899-8973-4181-afa5-c45aa5471013-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857524    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857538    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4635b899-8973-4181-afa5-c45aa5471013-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.857726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.857552    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ln2c\" (UniqueName: \"kubernetes.io/projected/2a77e321-f28b-4355-9187-b78c51baca4e-kube-api-access-2ln2c\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:03.988632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.988597    4549 generic.go:356] "Generic (PLEG): container finished" podID="2a77e321-f28b-4355-9187-b78c51baca4e" containerID="d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a" exitCode=0
Oct 31 15:14:03.988783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.988660    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"
Oct 31 15:14:03.988783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.988662    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" event={"ID":"2a77e321-f28b-4355-9187-b78c51baca4e","Type":"ContainerDied","Data":"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"}
Oct 31 15:14:03.988783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.988728    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b" event={"ID":"2a77e321-f28b-4355-9187-b78c51baca4e","Type":"ContainerDied","Data":"cef70f0504a8effcad5400c496aa353b7b6757a98d77ebcd69a76a9ac91468e5"}
Oct 31 15:14:03.988783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.988759    4549 scope.go:122] "RemoveContainer" containerID="d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"
Oct 31 15:14:03.990860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.990835    4549 generic.go:356] "Generic (PLEG): container finished" podID="4635b899-8973-4181-afa5-c45aa5471013" containerID="5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7" exitCode=0
Oct 31 15:14:03.990983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.990888    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"
Oct 31 15:14:03.990983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.990912    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerDied","Data":"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"}
Oct 31 15:14:03.991104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.990972    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj" event={"ID":"4635b899-8973-4181-afa5-c45aa5471013","Type":"ContainerDied","Data":"c13e2c7bf3566bd96892da3fec0d978afb474848141cf17e43ac4dcdd01dc58b"}
Oct 31 15:14:03.992738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.992708    4549 generic.go:356] "Generic (PLEG): container finished" podID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerID="de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f" exitCode=0
Oct 31 15:14:03.992854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.992749    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"
Oct 31 15:14:03.992854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.992756    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" event={"ID":"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c","Type":"ContainerDied","Data":"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"}
Oct 31 15:14:03.992854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:03.992795    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx" event={"ID":"cf72c8fc-3726-4a90-b5a2-c0681ea96e7c","Type":"ContainerDied","Data":"e668262faac28256fe66dc4c6d4e34e8d4e18928835c0d820368df4f5c355602"}
Oct 31 15:14:04.001966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.001949    4549 scope.go:122] "RemoveContainer" containerID="d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"
Oct 31 15:14:04.002344 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:04.002250    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a\": container with ID starting with d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a not found: ID does not exist" containerID="d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"
Oct 31 15:14:04.002344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.002288    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a"} err="failed to get container status \"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a\": rpc error: code = NotFound desc = could not find container \"d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a\": container with ID starting with d7e0076756ac56807a7d2dd65bcfc0888a14d1b49eda338b4301d9754ab31d4a not found: ID does not exist"
Oct 31 15:14:04.002344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.002313    4549 scope.go:122] "RemoveContainer" containerID="5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"
Oct 31 15:14:04.012513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.012497    4549 scope.go:122] "RemoveContainer" containerID="ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"
Oct 31 15:14:04.025360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.025338    4549 scope.go:122] "RemoveContainer" containerID="5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"
Oct 31 15:14:04.025684 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:04.025663    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7\": container with ID starting with 5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7 not found: ID does not exist" containerID="5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"
Oct 31 15:14:04.025765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.025691    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7"} err="failed to get container status \"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7\": rpc error: code = NotFound desc = could not find container \"5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7\": container with ID starting with 5bd0dad92073e8d0f299781601e5c3957f38b11420e23c51132b04b0b2a03bc7 not found: ID does not exist"
Oct 31 15:14:04.025765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.025711    4549 scope.go:122] "RemoveContainer" containerID="ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"
Oct 31 15:14:04.025984 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:04.025967    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8\": container with ID starting with ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8 not found: ID does not exist" containerID="ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"
Oct 31 15:14:04.026037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.025991    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8"} err="failed to get container status \"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8\": rpc error: code = NotFound desc = could not find container \"ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8\": container with ID starting with ff976d27aa12a152aa38f456b7e96cca6fcf3d7d4b400fccf5ca02b04e1a85d8 not found: ID does not exist"
Oct 31 15:14:04.026037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.026007    4549 scope.go:122] "RemoveContainer" containerID="de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"
Oct 31 15:14:04.040209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.038968    4549 scope.go:122] "RemoveContainer" containerID="de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"
Oct 31 15:14:04.040209 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:04.039418    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f\": container with ID starting with de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f not found: ID does not exist" containerID="de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"
Oct 31 15:14:04.040209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.039473    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f"} err="failed to get container status \"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f\": rpc error: code = NotFound desc = could not find container \"de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f\": container with ID starting with de10398c80c21fb0923b107d9305486381cd874090b74abd0e2649457b9ae72f not found: ID does not exist"
Oct 31 15:14:04.150430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.150346    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:14:04.155462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.155438    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-mhr4b"]
Oct 31 15:14:04.263179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.263142    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:14:04.268634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.268604    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-9n8vj"]
Oct 31 15:14:04.376289 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.376256    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:14:04.379783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:04.379761    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-r9gdx"]
Oct 31 15:14:05.420342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:05.420312    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" path="/var/lib/kubelet/pods/2a77e321-f28b-4355-9187-b78c51baca4e/volumes"
Oct 31 15:14:05.420826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:05.420785    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4635b899-8973-4181-afa5-c45aa5471013" path="/var/lib/kubelet/pods/4635b899-8973-4181-afa5-c45aa5471013/volumes"
Oct 31 15:14:05.421206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:05.421192    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" path="/var/lib/kubelet/pods/cf72c8fc-3726-4a90-b5a2-c0681ea96e7c/volumes"
Oct 31 15:14:07.020103 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:07.020065    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:14:24.582759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.582721    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:14:24.583278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.582981    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" containerID="cri-o://a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2" gracePeriod=30
Oct 31 15:14:24.586781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.586754    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:14:24.586988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.586961    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" containerID="cri-o://c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a" gracePeriod=30
Oct 31 15:14:24.607402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.607372    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:14:24.607612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:24.607587    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" containerID="cri-o://be40f651f0280704ff9965ed364ef0a2b426e2f7aa4e433744859ec3044bb9ff" gracePeriod=30
Oct 31 15:14:26.779292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:26.779253    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:14:26.779782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:26.779539    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy" containerID="cri-o://792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde" gracePeriod=30
Oct 31 15:14:26.788266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:26.788238    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:14:26.788500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:26.788476    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy" containerID="cri-o://424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b" gracePeriod=30
Oct 31 15:14:27.274304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:27.274265    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:27.274495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:27.274328    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:27.982142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:27.982096    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:27.982666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:27.982164    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:28.104441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:28.104399    4549 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:28.104670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:28.104469    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:29.953137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:29.953112    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:14:29.980432 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:29.980410    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:14:30.053996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.053965    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.054189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054053    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.054189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054088    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.054189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054118    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.054349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054204    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.054349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054223    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.054349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054263    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.054349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054273    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054542    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054662    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054786    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054852    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054880    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054784    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054919    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054947    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.054973    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.055011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055004    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055047    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055090    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055135    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\") pod \"741386e2-83f2-4eb8-95cf-dd56528a944d\" (UID: \"741386e2-83f2-4eb8-95cf-dd56528a944d\") "
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055196    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\") pod \"2da3adf0-55ca-4426-b398-fd0116716784\" (UID: \"2da3adf0-55ca-4426-b398-fd0116716784\") "
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055220    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.055878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055565    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:30.056199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055906    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.056199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.055911    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.056199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056082    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.056322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056294    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.056391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056350    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.056391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056369    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/741386e2-83f2-4eb8-95cf-dd56528a944d-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.056391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056385    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.056543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056400    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da3adf0-55ca-4426-b398-fd0116716784-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.056543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056415    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.056543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.056430    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.057255 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.057227    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.057255 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.057235    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.057940 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.057920    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "kube-api-access-vwxhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.058165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.058142    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:30.058383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.058359    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.058719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.058703    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:30.058805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.058783    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh" pod "741386e2-83f2-4eb8-95cf-dd56528a944d" (UID: "741386e2-83f2-4eb8-95cf-dd56528a944d"). InnerVolumeSpecName "kube-api-access-4q7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.059149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.059134    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token" pod "2da3adf0-55ca-4426-b398-fd0116716784" (UID: "2da3adf0-55ca-4426-b398-fd0116716784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.158036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158010    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxhv\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-kube-api-access-vwxhv\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158036    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158046    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158055    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158063    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158071    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158080    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da3adf0-55ca-4426-b398-fd0116716784-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158088    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q7xh\" (UniqueName: \"kubernetes.io/projected/741386e2-83f2-4eb8-95cf-dd56528a944d-kube-api-access-4q7xh\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158097    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158105    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da3adf0-55ca-4426-b398-fd0116716784-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158113    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da3adf0-55ca-4426-b398-fd0116716784-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.158204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.158123    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/741386e2-83f2-4eb8-95cf-dd56528a944d-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.299103 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.298563    4549 generic.go:356] "Generic (PLEG): container finished" podID="0d4074d5-50df-4721-9fc5-9460494d843f" containerID="be40f651f0280704ff9965ed364ef0a2b426e2f7aa4e433744859ec3044bb9ff" exitCode=0
Oct 31 15:14:30.299103 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.298739    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" event={"ID":"0d4074d5-50df-4721-9fc5-9460494d843f","Type":"ContainerDied","Data":"be40f651f0280704ff9965ed364ef0a2b426e2f7aa4e433744859ec3044bb9ff"}
Oct 31 15:14:30.300349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.300325    4549 generic.go:356] "Generic (PLEG): container finished" podID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerID="c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a" exitCode=0
Oct 31 15:14:30.300533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.300514    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" event={"ID":"741386e2-83f2-4eb8-95cf-dd56528a944d","Type":"ContainerDied","Data":"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"}
Oct 31 15:14:30.300661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.300541    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"
Oct 31 15:14:30.300661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.300557    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq" event={"ID":"741386e2-83f2-4eb8-95cf-dd56528a944d","Type":"ContainerDied","Data":"9d53ec12c84c9a6dc04f40af404b11a6a933312e452ded1fd108732dbef53dfa"}
Oct 31 15:14:30.300661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.300602    4549 scope.go:122] "RemoveContainer" containerID="c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"
Oct 31 15:14:30.306259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.305139    4549 generic.go:356] "Generic (PLEG): container finished" podID="2da3adf0-55ca-4426-b398-fd0116716784" containerID="a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2" exitCode=0
Oct 31 15:14:30.306259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.305184    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" event={"ID":"2da3adf0-55ca-4426-b398-fd0116716784","Type":"ContainerDied","Data":"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"}
Oct 31 15:14:30.306259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.305220    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh" event={"ID":"2da3adf0-55ca-4426-b398-fd0116716784","Type":"ContainerDied","Data":"74e85f6e6da3c1386791822d43af9b0a727ca98ecd626d54eed8205afcbdaa97"}
Oct 31 15:14:30.306259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.305299    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"
Oct 31 15:14:30.317376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.317357    4549 scope.go:122] "RemoveContainer" containerID="c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"
Oct 31 15:14:30.317692 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:30.317669    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a\": container with ID starting with c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a not found: ID does not exist" containerID="c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"
Oct 31 15:14:30.317783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.317704    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a"} err="failed to get container status \"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a\": rpc error: code = NotFound desc = could not find container \"c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a\": container with ID starting with c9cfea33aa869628f596001dcb205a12416e3d3c0fd476025c785e77024a640a not found: ID does not exist"
Oct 31 15:14:30.317783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.317727    4549 scope.go:122] "RemoveContainer" containerID="a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"
Oct 31 15:14:30.321833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.321812    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:14:30.327241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.327224    4549 scope.go:122] "RemoveContainer" containerID="a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"
Oct 31 15:14:30.327504 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:30.327488    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2\": container with ID starting with a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2 not found: ID does not exist" containerID="a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"
Oct 31 15:14:30.327600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.327517    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2"} err="failed to get container status \"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2\": rpc error: code = NotFound desc = could not find container \"a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2\": container with ID starting with a7cf4616526975cd131ac040a3ef2d7a791e1b947b5addfb5805f2a37df86dd2 not found: ID does not exist"
Oct 31 15:14:30.464248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464166    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464222    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464259    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464298    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464432    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464497    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464513    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.464687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464534    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464705    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.464944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464816    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\") pod \"0d4074d5-50df-4721-9fc5-9460494d843f\" (UID: \"0d4074d5-50df-4721-9fc5-9460494d843f\") "
Oct 31 15:14:30.465072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.464963    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:30.465377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465323    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.465377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465347    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.465657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465631    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.465943 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465917    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.466034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465947    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.466034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465962    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.466034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465977    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d4074d5-50df-4721-9fc5-9460494d843f-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.466034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.465994    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.468227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.468189    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "kube-api-access-vxw84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.472011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.469389    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:30.472011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.469622    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:30.472011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.470815    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy" pod "0d4074d5-50df-4721-9fc5-9460494d843f" (UID: "0d4074d5-50df-4721-9fc5-9460494d843f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:30.566870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.566835    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.566870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.566864    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d4074d5-50df-4721-9fc5-9460494d843f-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.566870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.566874    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxw84\" (UniqueName: \"kubernetes.io/projected/0d4074d5-50df-4721-9fc5-9460494d843f-kube-api-access-vxw84\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.567109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.566883    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d4074d5-50df-4721-9fc5-9460494d843f-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:30.583000 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.582958    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:14:30.586382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.586352    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-g74hh"]
Oct 31 15:14:30.694244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.694196    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:14:30.697947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:30.697923    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-48vdq"]
Oct 31 15:14:31.317820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.317787    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"
Oct 31 15:14:31.318281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.317794    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v" event={"ID":"0d4074d5-50df-4721-9fc5-9460494d843f","Type":"ContainerDied","Data":"bc2eda3b4ce971516d71aee86fc0d304000a09308dedbc3e7e8fa09711f0c83f"}
Oct 31 15:14:31.318281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.317976    4549 scope.go:122] "RemoveContainer" containerID="be40f651f0280704ff9965ed364ef0a2b426e2f7aa4e433744859ec3044bb9ff"
Oct 31 15:14:31.434218 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.434188    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da3adf0-55ca-4426-b398-fd0116716784" path="/var/lib/kubelet/pods/2da3adf0-55ca-4426-b398-fd0116716784/volumes"
Oct 31 15:14:31.434735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.434721    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" path="/var/lib/kubelet/pods/741386e2-83f2-4eb8-95cf-dd56528a944d/volumes"
Oct 31 15:14:31.489104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.489069    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:14:31.538433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:31.538400    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-rz94v"]
Oct 31 15:14:32.107726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.107703    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:14:32.111489 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.111471    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:14:32.189105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189074    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.189303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189164    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.189303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189218    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.189303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189255    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.189303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189283    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.189829    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190086    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190158    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190198    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190211    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190257    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190388    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190431    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190437    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190472    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190502    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190519    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190626    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.190654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190702    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190739    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190777    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\") pod \"d930d91f-40e7-40b4-a2e5-43908704cbcd\" (UID: \"d930d91f-40e7-40b4-a2e5-43908704cbcd\") "
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190820    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190713    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190747    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190895    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.191677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.191239    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.192382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192360    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:32.192382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.190915    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\") pod \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\" (UID: \"6824f94f-203d-4f4b-84ba-a9c6a81f736a\") "
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192908    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192931    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192946    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192957    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192966    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192974    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192983    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d930d91f-40e7-40b4-a2e5-43908704cbcd-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192991    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.192999    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.193008    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.193880 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.193841    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "kube-api-access-n9vfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:32.194069 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.194034    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:32.195228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.195181    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.195228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.195191    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:14:32.195382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.195225    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:32.195502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.195480    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "kube-api-access-49rnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:32.195618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.195479    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token" pod "6824f94f-203d-4f4b-84ba-a9c6a81f736a" (UID: "6824f94f-203d-4f4b-84ba-a9c6a81f736a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:32.196380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.196358    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy" pod "d930d91f-40e7-40b4-a2e5-43908704cbcd" (UID: "d930d91f-40e7-40b4-a2e5-43908704cbcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:14:32.294728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294690    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294723    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rnf\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-kube-api-access-49rnf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294736    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294746    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294754    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294762    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6824f94f-203d-4f4b-84ba-a9c6a81f736a-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294770    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d930d91f-40e7-40b4-a2e5-43908704cbcd-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.294957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.294779    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vfx\" (UniqueName: \"kubernetes.io/projected/d930d91f-40e7-40b4-a2e5-43908704cbcd-kube-api-access-n9vfx\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:32.328315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.328223    4549 generic.go:356] "Generic (PLEG): container finished" podID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerID="792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde" exitCode=0
Oct 31 15:14:32.328315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.328290    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"
Oct 31 15:14:32.328315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.328300    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" event={"ID":"d930d91f-40e7-40b4-a2e5-43908704cbcd","Type":"ContainerDied","Data":"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"}
Oct 31 15:14:32.328914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.328359    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29" event={"ID":"d930d91f-40e7-40b4-a2e5-43908704cbcd","Type":"ContainerDied","Data":"877934fa0ed0567bbe962f982f15c3391e36083e0730bd303a8a0128dbfb1aa5"}
Oct 31 15:14:32.328914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.328392    4549 scope.go:122] "RemoveContainer" containerID="792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"
Oct 31 15:14:32.332354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.332329    4549 generic.go:356] "Generic (PLEG): container finished" podID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerID="424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b" exitCode=0
Oct 31 15:14:32.332502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.332385    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"
Oct 31 15:14:32.332502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.332401    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" event={"ID":"6824f94f-203d-4f4b-84ba-a9c6a81f736a","Type":"ContainerDied","Data":"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"}
Oct 31 15:14:32.332502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.332444    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2" event={"ID":"6824f94f-203d-4f4b-84ba-a9c6a81f736a","Type":"ContainerDied","Data":"2f0f59abae742eb2bd2d883619c0dbe68ff1fbad7a99406a6ba1fa17f1515f99"}
Oct 31 15:14:32.356198 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.356177    4549 scope.go:122] "RemoveContainer" containerID="792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"
Oct 31 15:14:32.358829 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:32.356590    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde\": container with ID starting with 792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde not found: ID does not exist" containerID="792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"
Oct 31 15:14:32.358829 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.356630    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde"} err="failed to get container status \"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde\": rpc error: code = NotFound desc = could not find container \"792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde\": container with ID starting with 792004967aba5c05110e8e3b8c0959903ccc35803027ccda07359087ab802cde not found: ID does not exist"
Oct 31 15:14:32.358829 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.356654    4549 scope.go:122] "RemoveContainer" containerID="424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"
Oct 31 15:14:32.374878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.374852    4549 scope.go:122] "RemoveContainer" containerID="424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"
Oct 31 15:14:32.375314 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:32.375292    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b\": container with ID starting with 424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b not found: ID does not exist" containerID="424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"
Oct 31 15:14:32.375404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.375330    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b"} err="failed to get container status \"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b\": rpc error: code = NotFound desc = could not find container \"424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b\": container with ID starting with 424b9cfe79c2eb44c44c89d961c7c539871099a0af3b26c0e44b59a55255cf7b not found: ID does not exist"
Oct 31 15:14:32.567529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.567495    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:14:32.571426 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.571399    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-tkh29"]
Oct 31 15:14:32.680036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.679950    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:14:32.686036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:32.686012    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-7mmv2"]
Oct 31 15:14:33.420477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:33.420448    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" path="/var/lib/kubelet/pods/0d4074d5-50df-4721-9fc5-9460494d843f/volumes"
Oct 31 15:14:33.420984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:33.420969    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" path="/var/lib/kubelet/pods/6824f94f-203d-4f4b-84ba-a9c6a81f736a/volumes"
Oct 31 15:14:33.421434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:33.421412    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" path="/var/lib/kubelet/pods/d930d91f-40e7-40b4-a2e5-43908704cbcd/volumes"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.724200    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725131    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725151    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725174    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725182    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725192    4549 cpu_manager.go:414] "containerMap: removing container" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725199    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725214    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725222    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725230    4549 cpu_manager.go:414] "containerMap: removing container" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725236    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725246    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725254    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725268    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725275    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725305    4549 cpu_manager.go:414] "containerMap: removing container" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725312    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725324    4549 cpu_manager.go:414] "containerMap: removing container" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725331    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725343    4549 cpu_manager.go:414] "containerMap: removing container" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725349    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725508    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725524    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da3adf0-55ca-4426-b398-fd0116716784" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725534    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d930d91f-40e7-40b4-a2e5-43908704cbcd" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725544    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a77e321-f28b-4355-9187-b78c51baca4e" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725552    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="741386e2-83f2-4eb8-95cf-dd56528a944d" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725561    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6824f94f-203d-4f4b-84ba-a9c6a81f736a" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725591    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4074d5-50df-4721-9fc5-9460494d843f" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725600    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf72c8fc-3726-4a90-b5a2-c0681ea96e7c" containerName="istio-proxy"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725610    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="469f9541-1893-4d0a-899a-698108d2c6e4" containerName="main"
Oct 31 15:14:36.730594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.725618    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4635b899-8973-4181-afa5-c45aa5471013" containerName="istio-proxy"
Oct 31 15:14:36.732254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.731328    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:36.747171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.737797    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:14:36.783763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.783724    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f76cce\""
Oct 31 15:14:36.833151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.833122    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f76cce-dockercfg-r99w6\""
Oct 31 15:14:36.865817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.865786    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxll\" (UniqueName: \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:36.865977 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.865839    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:36.966999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.966962    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxll\" (UniqueName: \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:36.967154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.967013    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:36.967700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:36.967683    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:37.012906 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:37.012827    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:14:37.025504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:37.025477    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxll\" (UniqueName: \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\") pod \"payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:37.086285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:37.086256    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:37.472723 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:14:37.472685    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af3aff_aef4_4c79_a9e1_40a63f6b806f.slice/crio-a8c6d105f92ad42ba81257d7ec5a7e0eb9c2bc1885ccd8f6f18f83ba21e05822 WatchSource:0}: Error finding container a8c6d105f92ad42ba81257d7ec5a7e0eb9c2bc1885ccd8f6f18f83ba21e05822: Status 404 returned error can't find the container with id a8c6d105f92ad42ba81257d7ec5a7e0eb9c2bc1885ccd8f6f18f83ba21e05822
Oct 31 15:14:37.479124 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:37.479097    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:14:38.377745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.377713    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" event={"ID":"57af3aff-aef4-4c79-a9e1-40a63f6b806f","Type":"ContainerStarted","Data":"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"}
Oct 31 15:14:38.378225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.377768    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" event={"ID":"57af3aff-aef4-4c79-a9e1-40a63f6b806f","Type":"ContainerStarted","Data":"a8c6d105f92ad42ba81257d7ec5a7e0eb9c2bc1885ccd8f6f18f83ba21e05822"}
Oct 31 15:14:38.378225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.377798    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:38.490596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.490514    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" podStartSLOduration=2.490501406 podStartE2EDuration="2.490501406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:38.489828074 +0000 UTC m=+1719.216109076" watchObservedRunningTime="2026-10-31 15:14:38.490501406 +0000 UTC m=+1719.216782383"
Oct 31 15:14:38.772191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.771933    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:14:38.772191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:38.772142    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" podUID="ccfe8492-51bd-4a3e-aff9-255dbecee653" containerName="maas-api" containerID="cri-o://b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02" gracePeriod=30
Oct 31 15:14:39.269733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.269704    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:14:39.370460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.370371    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\") pod \"ccfe8492-51bd-4a3e-aff9-255dbecee653\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") "
Oct 31 15:14:39.370460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.370451    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\" (UniqueName: \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\") pod \"ccfe8492-51bd-4a3e-aff9-255dbecee653\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") "
Oct 31 15:14:39.370738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.370659    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\") pod \"ccfe8492-51bd-4a3e-aff9-255dbecee653\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") "
Oct 31 15:14:39.370796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.370760    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") pod \"ccfe8492-51bd-4a3e-aff9-255dbecee653\" (UID: \"ccfe8492-51bd-4a3e-aff9-255dbecee653\") "
Oct 31 15:14:39.370796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.370758    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca" pod "ccfe8492-51bd-4a3e-aff9-255dbecee653" (UID: "ccfe8492-51bd-4a3e-aff9-255dbecee653"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:14:39.371253 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.371223    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccfe8492-51bd-4a3e-aff9-255dbecee653-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:39.372837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.372810    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv" pod "ccfe8492-51bd-4a3e-aff9-255dbecee653" (UID: "ccfe8492-51bd-4a3e-aff9-255dbecee653"). InnerVolumeSpecName "kube-api-access-8v9xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:14:39.373020 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.373001    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls" pod "ccfe8492-51bd-4a3e-aff9-255dbecee653" (UID: "ccfe8492-51bd-4a3e-aff9-255dbecee653"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:14:39.373327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.373304    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs" pod "ccfe8492-51bd-4a3e-aff9-255dbecee653" (UID: "ccfe8492-51bd-4a3e-aff9-255dbecee653"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:14:39.391477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.391432    4549 generic.go:356] "Generic (PLEG): container finished" podID="ccfe8492-51bd-4a3e-aff9-255dbecee653" containerID="b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02" exitCode=0
Oct 31 15:14:39.391946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.391499    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"
Oct 31 15:14:39.391946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.391654    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" event={"ID":"ccfe8492-51bd-4a3e-aff9-255dbecee653","Type":"ContainerDied","Data":"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"}
Oct 31 15:14:39.391946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.391719    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr" event={"ID":"ccfe8492-51bd-4a3e-aff9-255dbecee653","Type":"ContainerDied","Data":"00614804dd2d94e8258ee902724daa58d1df7511ecf1053bf7c7d46b5da4f7c5"}
Oct 31 15:14:39.391946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.391751    4549 scope.go:122] "RemoveContainer" containerID="b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"
Oct 31 15:14:39.413133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.413110    4549 scope.go:122] "RemoveContainer" containerID="b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"
Oct 31 15:14:39.413492 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:14:39.413467    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02\": container with ID starting with b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02 not found: ID does not exist" containerID="b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"
Oct 31 15:14:39.413605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.413504    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02"} err="failed to get container status \"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02\": rpc error: code = NotFound desc = could not find container \"b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02\": container with ID starting with b8cfc8f22424cb55c2897ad61802bfdc327f235972c1c01655439b4303e7fa02 not found: ID does not exist"
Oct 31 15:14:39.472402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.472369    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:39.472402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.472398    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccfe8492-51bd-4a3e-aff9-255dbecee653-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:39.472665 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.472413    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9xv\" (UniqueName: \"kubernetes.io/projected/ccfe8492-51bd-4a3e-aff9-255dbecee653-kube-api-access-8v9xv\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:14:39.556988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.556954    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:14:39.560550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:39.560519    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-aac61b-65c77c8d96-bwfdr"]
Oct 31 15:14:40.097417 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.097379    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:14:40.097957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.097942    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ccfe8492-51bd-4a3e-aff9-255dbecee653" containerName="maas-api"
Oct 31 15:14:40.098028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.097959    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfe8492-51bd-4a3e-aff9-255dbecee653" containerName="maas-api"
Oct 31 15:14:40.098098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.098089    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfe8492-51bd-4a3e-aff9-255dbecee653" containerName="maas-api"
Oct 31 15:14:40.102680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.102664    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.155108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.155082    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 15:14:40.155108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.155082    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 15:14:40.155293 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.155113    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 15:14:40.155293 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.155153    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\""
Oct 31 15:14:40.181092 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181057    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181127    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181195    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181256    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181311    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181358    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.181418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.181387    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.212872 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.209524    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:14:40.282162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282127    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282186    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282231    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282311    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282344    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282383    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.282632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.282559    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.283456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.283425    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.284956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.284929    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.285061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.284929    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.285061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.284986    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.285306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.285286    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.339501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.339464    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.339501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.339499    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hlqx5\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.415543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.415456    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:40.851446 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:14:40.851413    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47f7af3_ef42_438f_a48d_8faa76f9b823.slice/crio-57ecaf4a4334ee9ed850adee8c1c2def497048e730224b79a6e12d7e811f19d7 WatchSource:0}: Error finding container 57ecaf4a4334ee9ed850adee8c1c2def497048e730224b79a6e12d7e811f19d7: Status 404 returned error can't find the container with id 57ecaf4a4334ee9ed850adee8c1c2def497048e730224b79a6e12d7e811f19d7
Oct 31 15:14:40.851855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.851837    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:14:40.853701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:40.853679    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:14:41.406887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:41.406850    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" event={"ID":"d47f7af3-ef42-438f-a48d-8faa76f9b823","Type":"ContainerStarted","Data":"57ecaf4a4334ee9ed850adee8c1c2def497048e730224b79a6e12d7e811f19d7"}
Oct 31 15:14:41.419179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:41.419150    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfe8492-51bd-4a3e-aff9-255dbecee653" path="/var/lib/kubelet/pods/ccfe8492-51bd-4a3e-aff9-255dbecee653/volumes"
Oct 31 15:14:44.724337 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:44.724305    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:14:44.724750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:44.724371    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:14:45.440343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:45.440306    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" event={"ID":"d47f7af3-ef42-438f-a48d-8faa76f9b823","Type":"ContainerStarted","Data":"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"}
Oct 31 15:14:45.440562 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:45.440369    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:14:45.442664 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:45.442638    4549 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hlqx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:45.442743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:45.442689    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:45.650278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:45.650217    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" podStartSLOduration=1.781207151 podStartE2EDuration="5.650199991s" totalImagesPullingTime="3.86899284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:45.607368948 +0000 UTC m=+1726.333649927" watchObservedRunningTime="2026-10-31 15:14:45.650199991 +0000 UTC m=+1726.376480968"
Oct 31 15:14:46.449952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:46.449922    4549 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hlqx5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:14:46.450350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:46.449971    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:14:49.395481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:49.395446    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:14:49.450207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:14:49.450174    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:15:00.028472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.028425    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:15:00.029148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.028707    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy" containerID="cri-o://0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a" gracePeriod=30
Oct 31 15:15:00.076350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.076319    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:15:00.076558 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.076519    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy" containerID="cri-o://0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7" gracePeriod=30
Oct 31 15:15:00.176621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.169764    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:00.176621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.174789    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.194390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.194362    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:00.198808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.198781    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.200368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.200316    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:00.205146 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.205125    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.228413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.228385    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 15:15:00.269015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.266537    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:00.277131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.277094    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:00.278255 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278227    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8j5\" (UniqueName: \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.278589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278497    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnm55\" (UniqueName: \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.278698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278664    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.278769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278751    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zs8\" (UniqueName: \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.278860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278843    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.278990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.278946    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.282390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.282364    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:00.380376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380337    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8j5\" (UniqueName: \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.380595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380409    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnm55\" (UniqueName: \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.380595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380485    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.380595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380563    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zs8\" (UniqueName: \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.382362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380657    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.382362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.380691    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.382362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.381971    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.382500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.382466    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.383013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.382988    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.439139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.439106    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zs8\" (UniqueName: \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.439890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.439855    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8j5\" (UniqueName: \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\") pod \"maas-api-key-cleanup-29890995-bqmls\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.440025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.439991    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnm55\" (UniqueName: \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.492515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.492466    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:00.519049 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.519013    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:00.527900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.527868    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:00.914344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.914311    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:00.915038 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:15:00.915007    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0b4bc2_75bd_4d4d_abaa_45d95971778a.slice/crio-909be6d575b11a0183ff8bb3996558155888695371054b54687db9f0788566f8 WatchSource:0}: Error finding container 909be6d575b11a0183ff8bb3996558155888695371054b54687db9f0788566f8: Status 404 returned error can't find the container with id 909be6d575b11a0183ff8bb3996558155888695371054b54687db9f0788566f8
Oct 31 15:15:00.976873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:00.976842    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:00.976992 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:15:00.976886    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3736f1a4_7d11_4a9b_8a89_0e8307e1bee2.slice/crio-a095b93d5d4a9ba90ba503a3f919b1a1331387acd6c480157143bb10e3828e49 WatchSource:0}: Error finding container a095b93d5d4a9ba90ba503a3f919b1a1331387acd6c480157143bb10e3828e49: Status 404 returned error can't find the container with id a095b93d5d4a9ba90ba503a3f919b1a1331387acd6c480157143bb10e3828e49
Oct 31 15:15:01.015044 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.015022    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:01.101293 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:15:01.101262    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd551e989_777a_4261_ad21_ede728d62359.slice/crio-b50aeafb0da4885dd92d215b723b62cc8d1c8812ca0d505982b825caaaee493b WatchSource:0}: Error finding container b50aeafb0da4885dd92d215b723b62cc8d1c8812ca0d505982b825caaaee493b: Status 404 returned error can't find the container with id b50aeafb0da4885dd92d215b723b62cc8d1c8812ca0d505982b825caaaee493b
Oct 31 15:15:01.578934 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.578896    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerStarted","Data":"dbd57e444b3ea2ff558df73236b7c22d117aab32ea0d2aba18e46b01add019d4"}
Oct 31 15:15:01.579130 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.578960    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerStarted","Data":"a095b93d5d4a9ba90ba503a3f919b1a1331387acd6c480157143bb10e3828e49"}
Oct 31 15:15:01.580371 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.580348    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerStarted","Data":"b50aeafb0da4885dd92d215b723b62cc8d1c8812ca0d505982b825caaaee493b"}
Oct 31 15:15:01.581982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.581960    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerStarted","Data":"f63785791e794727d0e0965fd4047d508a74b748f2a0872607e62afab191a936"}
Oct 31 15:15:01.582080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.582007    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerStarted","Data":"909be6d575b11a0183ff8bb3996558155888695371054b54687db9f0788566f8"}
Oct 31 15:15:01.695587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.695525    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" podStartSLOduration=1.695510574 podStartE2EDuration="1.695510574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:01.691026921 +0000 UTC m=+1742.417307904" watchObservedRunningTime="2026-10-31 15:15:01.695510574 +0000 UTC m=+1742.421791558"
Oct 31 15:15:01.803752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:01.803688    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" podStartSLOduration=1.803667918 podStartE2EDuration="1.803667918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:01.802437649 +0000 UTC m=+1742.528718641" watchObservedRunningTime="2026-10-31 15:15:01.803667918 +0000 UTC m=+1742.529948994"
Oct 31 15:15:02.590718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.590623    4549 generic.go:356] "Generic (PLEG): container finished" podID="d551e989-777a-4261-ad21-ede728d62359" containerID="ea1c513ca5374f2cac2667014950d38a57c249cdb96aee3d0a78e7602691aafe" exitCode=60
Oct 31 15:15:02.591171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.590716    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerDied","Data":"ea1c513ca5374f2cac2667014950d38a57c249cdb96aee3d0a78e7602691aafe"}
Oct 31 15:15:02.591171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.590990    4549 scope.go:122] "RemoveContainer" containerID="ea1c513ca5374f2cac2667014950d38a57c249cdb96aee3d0a78e7602691aafe"
Oct 31 15:15:02.592374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.592355    4549 generic.go:356] "Generic (PLEG): container finished" podID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerID="f63785791e794727d0e0965fd4047d508a74b748f2a0872607e62afab191a936" exitCode=60
Oct 31 15:15:02.592481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.592438    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerDied","Data":"f63785791e794727d0e0965fd4047d508a74b748f2a0872607e62afab191a936"}
Oct 31 15:15:02.592687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.592663    4549 scope.go:122] "RemoveContainer" containerID="f63785791e794727d0e0965fd4047d508a74b748f2a0872607e62afab191a936"
Oct 31 15:15:02.594362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.594271    4549 generic.go:356] "Generic (PLEG): container finished" podID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerID="dbd57e444b3ea2ff558df73236b7c22d117aab32ea0d2aba18e46b01add019d4" exitCode=60
Oct 31 15:15:02.594362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.594321    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerDied","Data":"dbd57e444b3ea2ff558df73236b7c22d117aab32ea0d2aba18e46b01add019d4"}
Oct 31 15:15:02.594628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:02.594537    4549 scope.go:122] "RemoveContainer" containerID="dbd57e444b3ea2ff558df73236b7c22d117aab32ea0d2aba18e46b01add019d4"
Oct 31 15:15:03.604992 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.604963    4549 generic.go:356] "Generic (PLEG): container finished" podID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerID="02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8" exitCode=60
Oct 31 15:15:03.605399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.605033    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerDied","Data":"02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8"}
Oct 31 15:15:03.605399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.605094    4549 scope.go:122] "RemoveContainer" containerID="dbd57e444b3ea2ff558df73236b7c22d117aab32ea0d2aba18e46b01add019d4"
Oct 31 15:15:03.605399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.605382    4549 scope.go:122] "RemoveContainer" containerID="02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8"
Oct 31 15:15:03.605757 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:03.605735    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl_odh-ai-gateway-infra(3736f1a4-7d11-4a9b-8a89-0e8307e1bee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2"
Oct 31 15:15:03.607200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.607180    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerStarted","Data":"a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"}
Oct 31 15:15:03.607480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.607465    4549 scope.go:122] "RemoveContainer" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"
Oct 31 15:15:03.607866 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:03.607845    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76_odh-ai-gateway-infra(d551e989-777a-4261-ad21-ede728d62359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" podUID="d551e989-777a-4261-ad21-ede728d62359"
Oct 31 15:15:03.608900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.608878    4549 generic.go:356] "Generic (PLEG): container finished" podID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerID="203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d" exitCode=60
Oct 31 15:15:03.608995 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.608953    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerDied","Data":"203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d"}
Oct 31 15:15:03.609207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.609191    4549 scope.go:122] "RemoveContainer" containerID="203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d"
Oct 31 15:15:03.609465 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:03.609446    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890995-bqmls_odh-ai-gateway-infra(dd0b4bc2-75bd-4d4d-abaa-45d95971778a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a"
Oct 31 15:15:03.706690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:03.706669    4549 scope.go:122] "RemoveContainer" containerID="f63785791e794727d0e0965fd4047d508a74b748f2a0872607e62afab191a936"
Oct 31 15:15:04.620193 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.620164    4549 scope.go:122] "RemoveContainer" containerID="02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8"
Oct 31 15:15:04.620690 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:04.620532    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl_odh-ai-gateway-infra(3736f1a4-7d11-4a9b-8a89-0e8307e1bee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2"
Oct 31 15:15:04.621900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.621878    4549 generic.go:356] "Generic (PLEG): container finished" podID="d551e989-777a-4261-ad21-ede728d62359" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36" exitCode=60
Oct 31 15:15:04.622013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.621935    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerDied","Data":"a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"}
Oct 31 15:15:04.622013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.621987    4549 scope.go:122] "RemoveContainer" containerID="ea1c513ca5374f2cac2667014950d38a57c249cdb96aee3d0a78e7602691aafe"
Oct 31 15:15:04.622357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.622340    4549 scope.go:122] "RemoveContainer" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"
Oct 31 15:15:04.622650 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:04.622627    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76_odh-ai-gateway-infra(d551e989-777a-4261-ad21-ede728d62359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" podUID="d551e989-777a-4261-ad21-ede728d62359"
Oct 31 15:15:04.623932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:04.623917    4549 scope.go:122] "RemoveContainer" containerID="203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d"
Oct 31 15:15:04.624129 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:04.624114    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890995-bqmls_odh-ai-gateway-infra(dd0b4bc2-75bd-4d4d-abaa-45d95971778a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a"
Oct 31 15:15:05.356495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.356463    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:15:05.405215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.405188    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:15:05.457620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457556    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.457816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457664    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.457816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457702    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.457816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457756    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.457816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457796    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457826    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.458027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457845    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457884    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457916    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.457981    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.458282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458064    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458077    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.458282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458104    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458146    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.458282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458238    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458290    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458326    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458332    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458368    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458416    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458450    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\") pod \"09136301-7149-4162-a5db-8b469f6ba404\" (UID: \"09136301-7149-4162-a5db-8b469f6ba404\") "
Oct 31 15:15:05.458542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458537    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.458950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.458564    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\") pod \"821afb3c-91c1-4936-be5b-ff18da551e27\" (UID: \"821afb3c-91c1-4936-be5b-ff18da551e27\") "
Oct 31 15:15:05.459118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.459091    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.459346 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.459307    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:05.459441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.459418    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.459510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.459441    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.459510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.459447    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460048    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821afb3c-91c1-4936-be5b-ff18da551e27-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460073    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460087    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460103    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460116    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460130    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460143    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460157    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09136301-7149-4162-a5db-8b469f6ba404-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460171    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460185    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.460948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.460746    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:15:05.461754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.461727    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:05.461847 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.461796    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.461906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.461889    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "kube-api-access-6s55b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:05.461989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.461908    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy" pod "821afb3c-91c1-4936-be5b-ff18da551e27" (UID: "821afb3c-91c1-4936-be5b-ff18da551e27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:05.462301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.462283    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "kube-api-access-nqpkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:05.462301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.462291    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:15:05.462383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.462364    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token" pod "09136301-7149-4162-a5db-8b469f6ba404" (UID: "09136301-7149-4162-a5db-8b469f6ba404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.561956    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09136301-7149-4162-a5db-8b469f6ba404-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.561985    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821afb3c-91c1-4936-be5b-ff18da551e27-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.561996    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821afb3c-91c1-4936-be5b-ff18da551e27-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.562006    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.562015    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpkd\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-kube-api-access-nqpkd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.562024    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09136301-7149-4162-a5db-8b469f6ba404-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.562033    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s55b\" (UniqueName: \"kubernetes.io/projected/821afb3c-91c1-4936-be5b-ff18da551e27-kube-api-access-6s55b\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.562037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.562043    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09136301-7149-4162-a5db-8b469f6ba404-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:05.635948 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.635920    4549 scope.go:122] "RemoveContainer" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"
Oct 31 15:15:05.636458 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:05.636285    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76_odh-ai-gateway-infra(d551e989-777a-4261-ad21-ede728d62359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" podUID="d551e989-777a-4261-ad21-ede728d62359"
Oct 31 15:15:05.637280 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.637249    4549 generic.go:356] "Generic (PLEG): container finished" podID="821afb3c-91c1-4936-be5b-ff18da551e27" containerID="0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7" exitCode=0
Oct 31 15:15:05.637421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.637311    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"
Oct 31 15:15:05.637421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.637336    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" event={"ID":"821afb3c-91c1-4936-be5b-ff18da551e27","Type":"ContainerDied","Data":"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"}
Oct 31 15:15:05.637421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.637384    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx" event={"ID":"821afb3c-91c1-4936-be5b-ff18da551e27","Type":"ContainerDied","Data":"26d06399ab624fc5d0ae95682cd5eac74a7121378f89372f80fe0d9b571f982d"}
Oct 31 15:15:05.637623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.637485    4549 scope.go:122] "RemoveContainer" containerID="0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"
Oct 31 15:15:05.639371 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.639349    4549 generic.go:356] "Generic (PLEG): container finished" podID="09136301-7149-4162-a5db-8b469f6ba404" containerID="0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a" exitCode=0
Oct 31 15:15:05.639493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.639385    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" event={"ID":"09136301-7149-4162-a5db-8b469f6ba404","Type":"ContainerDied","Data":"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"}
Oct 31 15:15:05.639493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.639393    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"
Oct 31 15:15:05.639493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.639422    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md" event={"ID":"09136301-7149-4162-a5db-8b469f6ba404","Type":"ContainerDied","Data":"6b77c91f60df96e6e455a1bd13a677ff751e2a1c7f1da4e25d0d01473292a147"}
Oct 31 15:15:05.649634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.649383    4549 scope.go:122] "RemoveContainer" containerID="0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"
Oct 31 15:15:05.650106 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:05.650064    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7\": container with ID starting with 0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7 not found: ID does not exist" containerID="0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"
Oct 31 15:15:05.650212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.650111    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7"} err="failed to get container status \"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7\": rpc error: code = NotFound desc = could not find container \"0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7\": container with ID starting with 0d97f5254380ec8fd128ebcce56bb9e3bcd6eb80464f0d6f728e1b8b34f9c8d7 not found: ID does not exist"
Oct 31 15:15:05.650212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.650136    4549 scope.go:122] "RemoveContainer" containerID="0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"
Oct 31 15:15:05.664687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.664667    4549 scope.go:122] "RemoveContainer" containerID="0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"
Oct 31 15:15:05.664996 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:05.664977    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a\": container with ID starting with 0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a not found: ID does not exist" containerID="0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"
Oct 31 15:15:05.665050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.665006    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a"} err="failed to get container status \"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a\": rpc error: code = NotFound desc = could not find container \"0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a\": container with ID starting with 0298a691ea57ce5b9f9d1fe2939d1edf51031c73e8c4bbffba985b0802f37c9a not found: ID does not exist"
Oct 31 15:15:05.801496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.801404    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:15:05.805587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.805549    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x27md"]
Oct 31 15:15:05.916716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.916623    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:15:05.921042 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:05.921006    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-jrgbx"]
Oct 31 15:15:07.012591 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:07.012538    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:15:07.420154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:07.420110    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09136301-7149-4162-a5db-8b469f6ba404" path="/var/lib/kubelet/pods/09136301-7149-4162-a5db-8b469f6ba404/volumes"
Oct 31 15:15:07.420710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:07.420689    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" path="/var/lib/kubelet/pods/821afb3c-91c1-4936-be5b-ff18da551e27/volumes"
Oct 31 15:15:14.412713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:14.412625    4549 scope.go:122] "RemoveContainer" containerID="203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d"
Oct 31 15:15:14.728622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:14.728562    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerStarted","Data":"12ce36a35d8e54da4d2e79c228532806dc6302ada61e1ffb98174007b14e5084"}
Oct 31 15:15:15.775804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:15.775555    4549 generic.go:356] "Generic (PLEG): container finished" podID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerID="12ce36a35d8e54da4d2e79c228532806dc6302ada61e1ffb98174007b14e5084" exitCode=60
Oct 31 15:15:15.775804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:15.775608    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerDied","Data":"12ce36a35d8e54da4d2e79c228532806dc6302ada61e1ffb98174007b14e5084"}
Oct 31 15:15:15.775804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:15.775688    4549 scope.go:122] "RemoveContainer" containerID="203290d601782e2059bfaccc626732c581a2d0392b2620a64600eae14e75204d"
Oct 31 15:15:15.776525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:15.776220    4549 scope.go:122] "RemoveContainer" containerID="12ce36a35d8e54da4d2e79c228532806dc6302ada61e1ffb98174007b14e5084"
Oct 31 15:15:15.776816 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:15.776559    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890995-bqmls_odh-ai-gateway-infra(dd0b4bc2-75bd-4d4d-abaa-45d95971778a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a"
Oct 31 15:15:15.882956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:15.882924    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:16.412236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:16.412203    4549 scope.go:122] "RemoveContainer" containerID="02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8"
Oct 31 15:15:16.412444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:16.412260    4549 scope.go:122] "RemoveContainer" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"
Oct 31 15:15:16.792377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:16.792338    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerStarted","Data":"dae98d76864efb830bb5980efd8b1e64cfccf58f36d09f6fa22760d3b6e5c200"}
Oct 31 15:15:17.079613 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.079590    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:17.245712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.245687    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\" (UniqueName: \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\") pod \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") "
Oct 31 15:15:17.245864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.245761    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\") pod \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\" (UID: \"dd0b4bc2-75bd-4d4d-abaa-45d95971778a\") "
Oct 31 15:15:17.246564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.246537    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca" pod "dd0b4bc2-75bd-4d4d-abaa-45d95971778a" (UID: "dd0b4bc2-75bd-4d4d-abaa-45d95971778a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:17.248176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.248146    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5" pod "dd0b4bc2-75bd-4d4d-abaa-45d95971778a" (UID: "dd0b4bc2-75bd-4d4d-abaa-45d95971778a"). InnerVolumeSpecName "kube-api-access-4m8j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:17.347535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.347461    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8j5\" (UniqueName: \"kubernetes.io/projected/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-kube-api-access-4m8j5\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:17.347535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.347493    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd0b4bc2-75bd-4d4d-abaa-45d95971778a-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:17.801062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.801031    4549 generic.go:356] "Generic (PLEG): container finished" podID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerID="dae98d76864efb830bb5980efd8b1e64cfccf58f36d09f6fa22760d3b6e5c200" exitCode=60
Oct 31 15:15:17.801526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.801102    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerDied","Data":"dae98d76864efb830bb5980efd8b1e64cfccf58f36d09f6fa22760d3b6e5c200"}
Oct 31 15:15:17.801526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.801153    4549 scope.go:122] "RemoveContainer" containerID="02daa0dce2bc60f047da5f872e5b3b7fc98efac820012ecf80f0a3b4815447e8"
Oct 31 15:15:17.801659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.801541    4549 scope.go:122] "RemoveContainer" containerID="dae98d76864efb830bb5980efd8b1e64cfccf58f36d09f6fa22760d3b6e5c200"
Oct 31 15:15:17.801912 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:17.801888    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl_odh-ai-gateway-infra(3736f1a4-7d11-4a9b-8a89-0e8307e1bee2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2"
Oct 31 15:15:17.803125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.803106    4549 generic.go:356] "Generic (PLEG): container finished" podID="d551e989-777a-4261-ad21-ede728d62359" containerID="ec8357f324cc3502abe32b6b614627b2a29de74babcc09d22b4f7c4f571c562d" exitCode=60
Oct 31 15:15:17.803225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.803178    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerDied","Data":"ec8357f324cc3502abe32b6b614627b2a29de74babcc09d22b4f7c4f571c562d"}
Oct 31 15:15:17.803530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.803514    4549 scope.go:122] "RemoveContainer" containerID="ec8357f324cc3502abe32b6b614627b2a29de74babcc09d22b4f7c4f571c562d"
Oct 31 15:15:17.803857 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:17.803840    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76_odh-ai-gateway-infra(d551e989-777a-4261-ad21-ede728d62359)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" podUID="d551e989-777a-4261-ad21-ede728d62359"
Oct 31 15:15:17.804539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.804522    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"
Oct 31 15:15:17.804626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.804562    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls" event={"ID":"dd0b4bc2-75bd-4d4d-abaa-45d95971778a","Type":"ContainerDied","Data":"909be6d575b11a0183ff8bb3996558155888695371054b54687db9f0788566f8"}
Oct 31 15:15:17.813796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.813776    4549 scope.go:122] "RemoveContainer" containerID="a6bae049e1a21548f34dcc10c2e51a0066c8c4368d88f5e6efe86ef63bc62e36"
Oct 31 15:15:17.826242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:17.826223    4549 scope.go:122] "RemoveContainer" containerID="12ce36a35d8e54da4d2e79c228532806dc6302ada61e1ffb98174007b14e5084"
Oct 31 15:15:18.031748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:18.031714    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:18.035508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:18.035482    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:18.039011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:18.038982    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890995-bqmls"]
Oct 31 15:15:18.979821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:18.979793    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:19.068537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.068448    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\" (UniqueName: \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\") pod \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") "
Oct 31 15:15:19.068758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.068539    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\") pod \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\" (UID: \"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2\") "
Oct 31 15:15:19.068953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.068931    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca" pod "3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" (UID: "3736f1a4-7d11-4a9b-8a89-0e8307e1bee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:19.069153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.069137    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:19.070850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.070827    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8" pod "3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" (UID: "3736f1a4-7d11-4a9b-8a89-0e8307e1bee2"). InnerVolumeSpecName "kube-api-access-b4zs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:19.153476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.153442    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:19.170353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.170315    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4zs8\" (UniqueName: \"kubernetes.io/projected/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2-kube-api-access-b4zs8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:19.308549 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.308524    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:19.429925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.429840    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" path="/var/lib/kubelet/pods/dd0b4bc2-75bd-4d4d-abaa-45d95971778a/volumes"
Oct 31 15:15:19.474452 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.474417    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\" (UniqueName: \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\") pod \"d551e989-777a-4261-ad21-ede728d62359\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") "
Oct 31 15:15:19.474650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.474526    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\") pod \"d551e989-777a-4261-ad21-ede728d62359\" (UID: \"d551e989-777a-4261-ad21-ede728d62359\") "
Oct 31 15:15:19.474887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.474852    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca" pod "d551e989-777a-4261-ad21-ede728d62359" (UID: "d551e989-777a-4261-ad21-ede728d62359"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:19.475267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.475218    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d551e989-777a-4261-ad21-ede728d62359-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:19.476752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.476729    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55" pod "d551e989-777a-4261-ad21-ede728d62359" (UID: "d551e989-777a-4261-ad21-ede728d62359"). InnerVolumeSpecName "kube-api-access-mnm55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:19.576473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.576435    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnm55\" (UniqueName: \"kubernetes.io/projected/d551e989-777a-4261-ad21-ede728d62359-kube-api-access-mnm55\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:19.827592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.827483    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"
Oct 31 15:15:19.827592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.827499    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl" event={"ID":"3736f1a4-7d11-4a9b-8a89-0e8307e1bee2","Type":"ContainerDied","Data":"a095b93d5d4a9ba90ba503a3f919b1a1331387acd6c480157143bb10e3828e49"}
Oct 31 15:15:19.827592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.827564    4549 scope.go:122] "RemoveContainer" containerID="dae98d76864efb830bb5980efd8b1e64cfccf58f36d09f6fa22760d3b6e5c200"
Oct 31 15:15:19.829320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.829261    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76" event={"ID":"d551e989-777a-4261-ad21-ede728d62359","Type":"ContainerDied","Data":"b50aeafb0da4885dd92d215b723b62cc8d1c8812ca0d505982b825caaaee493b"}
Oct 31 15:15:19.829524 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.829510    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"
Oct 31 15:15:19.843738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.843705    4549 scope.go:122] "RemoveContainer" containerID="ec8357f324cc3502abe32b6b614627b2a29de74babcc09d22b4f7c4f571c562d"
Oct 31 15:15:19.992334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:19.992300    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:20.041394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:20.041363    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890995-spc76"]
Oct 31 15:15:20.106425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:20.106331    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:20.109785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:20.109759    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29890995-lgqnl"]
Oct 31 15:15:21.420786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:21.420751    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" path="/var/lib/kubelet/pods/3736f1a4-7d11-4a9b-8a89-0e8307e1bee2/volumes"
Oct 31 15:15:21.421357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:21.421341    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d551e989-777a-4261-ad21-ede728d62359" path="/var/lib/kubelet/pods/d551e989-777a-4261-ad21-ede728d62359/volumes"
Oct 31 15:15:31.668551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:31.668511    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:15:31.669030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:31.668812    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy" containerID="cri-o://b0db581002682e3663ada0463f81911803ab625b0d3aa6689a9af598b626b61c" gracePeriod=30
Oct 31 15:15:36.966370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:36.966334    4549 generic.go:356] "Generic (PLEG): container finished" podID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerID="b0db581002682e3663ada0463f81911803ab625b0d3aa6689a9af598b626b61c" exitCode=0
Oct 31 15:15:36.966799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:36.966408    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" event={"ID":"2c99068e-c7ce-48ba-a48c-07191442ca11","Type":"ContainerDied","Data":"b0db581002682e3663ada0463f81911803ab625b0d3aa6689a9af598b626b61c"}
Oct 31 15:15:36.966799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:36.966459    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2" event={"ID":"2c99068e-c7ce-48ba-a48c-07191442ca11","Type":"ContainerDied","Data":"d8072268534a05beb2c7c8120b0f345f494dc1c41a9e48305bde722a4f2152a5"}
Oct 31 15:15:36.966799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:36.966479    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8072268534a05beb2c7c8120b0f345f494dc1c41a9e48305bde722a4f2152a5"
Oct 31 15:15:36.978976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:36.978953    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:15:37.013545 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:15:37.013503    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:15:37.108917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.108883    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.108954    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.108986    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109053    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109104    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109151    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109206    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:37.109263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109235    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109294    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:37.109464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109367    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109441    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\") pod \"2c99068e-c7ce-48ba-a48c-07191442ca11\" (UID: \"2c99068e-c7ce-48ba-a48c-07191442ca11\") "
Oct 31 15:15:37.109649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109482    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:37.109649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109628    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:37.109753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109698    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:15:37.110250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.109992    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.110250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.110016    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.110250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.110032    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c99068e-c7ce-48ba-a48c-07191442ca11-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.110250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.110047    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.110250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.110139    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.111798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.111764    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:15:37.111993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.111970    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:37.112210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.112186    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:15:37.112381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.112363    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn" pod "2c99068e-c7ce-48ba-a48c-07191442ca11" (UID: "2c99068e-c7ce-48ba-a48c-07191442ca11"). InnerVolumeSpecName "kube-api-access-gt5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:15:37.211477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.211382    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.211477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.211423    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt5zn\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-kube-api-access-gt5zn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.211477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.211437    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.211477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.211450    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c99068e-c7ce-48ba-a48c-07191442ca11-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:15:37.973037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:37.973002    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"
Oct 31 15:15:38.187937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:38.187907    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:15:38.239382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:38.239276    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-mklf2"]
Oct 31 15:15:39.420501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:15:39.420466    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" path="/var/lib/kubelet/pods/2c99068e-c7ce-48ba-a48c-07191442ca11/volumes"
Oct 31 15:16:02.732777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:02.732750    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:16:02.773537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:02.773503    4549 scope.go:122] "RemoveContainer" containerID="b0db581002682e3663ada0463f81911803ab625b0d3aa6689a9af598b626b61c"
Oct 31 15:16:02.843959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:02.843926    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:16:03.844337 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.844298    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:16:03.845628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.845591    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.845789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.845776    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.845882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.845872    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.845985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.845974    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.846056 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.846048    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.846120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.846111    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.846190 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.846182    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.846965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.846862    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.847444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.847428    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.847552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.847539    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.847731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.847717    4549 cpu_manager.go:414] "containerMap: removing container" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy"
Oct 31 15:16:03.847819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.847807    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy"
Oct 31 15:16:03.847905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.847895    4549 cpu_manager.go:414] "containerMap: removing container" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy"
Oct 31 15:16:03.848327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.848307    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy"
Oct 31 15:16:03.848494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.848483    4549 cpu_manager.go:414] "containerMap: removing container" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy"
Oct 31 15:16:03.848605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.848591    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy"
Oct 31 15:16:03.848744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.848732    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.848824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.848813    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.849080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849067    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.849171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849161    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.849257 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849248    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.849331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849322    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:16:03.849405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849396    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.849480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849471    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.849556 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849547    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:16:03.849662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849651    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="821afb3c-91c1-4936-be5b-ff18da551e27" containerName="istio-proxy"
Oct 31 15:16:03.849738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849729    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c99068e-c7ce-48ba-a48c-07191442ca11" containerName="istio-proxy"
Oct 31 15:16:03.849818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849809    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="09136301-7149-4162-a5db-8b469f6ba404" containerName="istio-proxy"
Oct 31 15:16:03.849886 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.849878    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:16:03.864391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.856378    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.005593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:03.992810    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9ab5df-openshift-default-dockercfg-xkzc9\""
Oct 31 15:16:04.005593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.003634    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:16:04.008704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.006674    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.008704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.006777    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.008704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.006823    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmx2k\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.008704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.006853    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.008704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.006879    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.041870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.026071    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.041870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.026113    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.041870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.026200    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.041870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.026292    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127166    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127229    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127273    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127326    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127369    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmx2k\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127391    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127408    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127439    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.127640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.127457    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.128101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.128071    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.128207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.128170    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.128355 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.128327    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.128498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.128477    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.128925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.128898    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.130699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.130673    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.130825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.130806    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.184390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.184358    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.184611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.184420    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmx2k\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\") pod \"e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") " pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.202793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.202763    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:04.653807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.653775    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:16:04.692890 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:16:04.692858    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa34287_61c1_4d84_8d77_81cefe01dfde.slice/crio-0903a64a7557fd7af991794472eb8754e9594898aa4bc32df5b1b9056d9bbab4 WatchSource:0}: Error finding container 0903a64a7557fd7af991794472eb8754e9594898aa4bc32df5b1b9056d9bbab4: Status 404 returned error can't find the container with id 0903a64a7557fd7af991794472eb8754e9594898aa4bc32df5b1b9056d9bbab4
Oct 31 15:16:04.695500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.695468    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:16:04.695597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.695540    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:16:04.695642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:04.695603    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:16:05.176485 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:05.176446    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerStarted","Data":"9a297dea371488177f11b74708c03a83e9e8313306037075584cb7b336992f87"}
Oct 31 15:16:05.176928 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:05.176496    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerStarted","Data":"0903a64a7557fd7af991794472eb8754e9594898aa4bc32df5b1b9056d9bbab4"}
Oct 31 15:16:05.203840 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:05.203808    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:05.211464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:05.211425    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:05.294069 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:05.294017    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podStartSLOduration=2.2940017409999998 podStartE2EDuration="2.294001741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:05.292993305 +0000 UTC m=+1806.019274296" watchObservedRunningTime="2026-10-31 15:16:05.294001741 +0000 UTC m=+1806.020282718"
Oct 31 15:16:06.183479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:06.183442    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:06.184386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:16:06.184370    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:16:07.012864 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:16:07.012824    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:16:37.012937 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:16:37.012895    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:17:07.013907 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:17:07.013864    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:17:20.814776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:20.814744    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:17:20.815363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:20.814969    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" containerID="cri-o://7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f" gracePeriod=30
Oct 31 15:17:23.782466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:23.782423    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:17:23.783087 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:23.782479    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:17:26.105409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.105370    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:17:26.200157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200122    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200197    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200311    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200354    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200385    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200432    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200542    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.200737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200430    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:17:26.200737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200520    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:17:26.200737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200603    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:17:26.200955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200841    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.201002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200954    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:17:26.201002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.200970    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\") pod \"ac19db83-4bfe-4282-a380-e5143b46e48c\" (UID: \"ac19db83-4bfe-4282-a380-e5143b46e48c\") "
Oct 31 15:17:26.201410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201260    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:17:26.201770 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201747    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.201888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201775    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.201888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201789    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.201888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201803    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.201888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.201819    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac19db83-4bfe-4282-a380-e5143b46e48c-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.203517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.203493    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:17:26.203652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.203562    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "kube-api-access-mbbt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:17:26.203652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.203603    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:17:26.203747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.203715    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy" pod "ac19db83-4bfe-4282-a380-e5143b46e48c" (UID: "ac19db83-4bfe-4282-a380-e5143b46e48c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:17:26.303624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.303502    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbt7\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-kube-api-access-mbbt7\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.303624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.303539    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.303624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.303554    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.303624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.303585    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac19db83-4bfe-4282-a380-e5143b46e48c-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:17:26.861669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.861637    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerID="7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f" exitCode=0
Oct 31 15:17:26.861885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.861676    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" event={"ID":"ac19db83-4bfe-4282-a380-e5143b46e48c","Type":"ContainerDied","Data":"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"}
Oct 31 15:17:26.861885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.861757    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr" event={"ID":"ac19db83-4bfe-4282-a380-e5143b46e48c","Type":"ContainerDied","Data":"f9083c1c6758218a0c356ff4bd0e8c6b73e077b989701e1c9a9ec84f4dfcdce9"}
Oct 31 15:17:26.861885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.861701    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"
Oct 31 15:17:26.861885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.861788    4549 scope.go:122] "RemoveContainer" containerID="7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"
Oct 31 15:17:26.872909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.872887    4549 scope.go:122] "RemoveContainer" containerID="7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"
Oct 31 15:17:26.873164 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:17:26.873144    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f\": container with ID starting with 7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f not found: ID does not exist" containerID="7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"
Oct 31 15:17:26.873211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:26.873178    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f"} err="failed to get container status \"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f\": rpc error: code = NotFound desc = could not find container \"7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f\": container with ID starting with 7e1b1ac1484f5837ef127c6cdd478044451a254432592288e09e9036c2373b7f not found: ID does not exist"
Oct 31 15:17:27.075796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:27.075765    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:17:27.125353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:27.125269    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-dlzxr"]
Oct 31 15:17:27.427888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:27.427802    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" path="/var/lib/kubelet/pods/ac19db83-4bfe-4282-a380-e5143b46e48c/volumes"
Oct 31 15:17:37.014849 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:17:37.014808    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:17:49.047077 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.047047    4549 generic.go:356] "Generic (PLEG): container finished" podID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerID="9a297dea371488177f11b74708c03a83e9e8313306037075584cb7b336992f87" exitCode=0
Oct 31 15:17:49.047561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.047086    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerDied","Data":"9a297dea371488177f11b74708c03a83e9e8313306037075584cb7b336992f87"}
Oct 31 15:17:49.047561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.047447    4549 scope.go:122] "RemoveContainer" containerID="9a297dea371488177f11b74708c03a83e9e8313306037075584cb7b336992f87"
Oct 31 15:17:49.049224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.049184    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:17:49.049311 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.049278    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:17:49.049356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:49.049326    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:17:50.056693 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:50.056655    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerStarted","Data":"3eb0a4c70239cab9686538aae42f1506fc974f567782e46b6a48348957b39f70"}
Oct 31 15:17:50.207849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:50.207817    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:17:50.208363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:50.208339    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:50.208536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:50.208511    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:51.184215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:51.184178    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:17:51.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:51.209062    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:51.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:51.209120    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:52.224531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:52.220034    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:52.224531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:52.222806    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:53.203617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:53.203560    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:53.203936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:53.203904    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:54.206597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:54.205854    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:54.206597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:54.205925    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:55.204206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:55.204168    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Oct 31 15:17:55.204425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:55.204248    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Oct 31 15:17:56.207788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:56.207755    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:17:56.208969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:17:56.208946    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:18:03.054290 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054253    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054788    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054800    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054810    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054816    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736f1a4-7d11-4a9b-8a89-0e8307e1bee2" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054826    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054831    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0b4bc2-75bd-4d4d-abaa-45d95971778a" containerName="cleanup"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054838    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy"
Oct 31 15:18:03.054850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054843    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy"
Oct 31 15:18:03.055245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054933    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac19db83-4bfe-4282-a380-e5143b46e48c" containerName="istio-proxy"
Oct 31 15:18:03.055245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.054947    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d551e989-777a-4261-ad21-ede728d62359" containerName="cleanup"
Oct 31 15:18:03.059460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.059438    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.119025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.118934    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.119025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.118986    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.119251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.119119    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.119251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.119225    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bf5\" (UniqueName: \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.161556 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.161525    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9cc557-serving-cert\""
Oct 31 15:18:03.165226 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.165202    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:18:03.220827 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.220785    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.221019 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.220835    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.221019 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.220875    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.221019 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.220917    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bf5\" (UniqueName: \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.221242 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:18:03.221216    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-serving-cert: secret "maas-api-e2e-api-b-9cc557-serving-cert" not found
Oct 31 15:18:03.221331 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:18:03.221295    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls podName:9b7de416-7113-420c-96c0-23d9ad240d5c nodeName:}" failed. No retries permitted until 2026-10-31 15:18:03.72127322 +0000 UTC m=+1924.447554177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls") pod "maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" (UID: "9b7de416-7113-420c-96c0-23d9ad240d5c") : secret "maas-api-e2e-api-b-9cc557-serving-cert" not found
Oct 31 15:18:03.221938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.221832    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.224477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.224423    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.279794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.279767    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bf5\" (UniqueName: \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.727484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.727437    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.730052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.730018    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") pod \"maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:03.975355 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:03.975323    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:04.356539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:04.356514    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:18:04.357301 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:18:04.357275    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7de416_7113_420c_96c0_23d9ad240d5c.slice/crio-0ed44bfbfce5e82851379d8e5c6a66097a3edaf715698667c506f294a9e024c0 WatchSource:0}: Error finding container 0ed44bfbfce5e82851379d8e5c6a66097a3edaf715698667c506f294a9e024c0: Status 404 returned error can't find the container with id 0ed44bfbfce5e82851379d8e5c6a66097a3edaf715698667c506f294a9e024c0
Oct 31 15:18:05.230833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:05.230793    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" event={"ID":"9b7de416-7113-420c-96c0-23d9ad240d5c","Type":"ContainerStarted","Data":"0ed44bfbfce5e82851379d8e5c6a66097a3edaf715698667c506f294a9e024c0"}
Oct 31 15:18:06.240873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:06.240840    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" event={"ID":"9b7de416-7113-420c-96c0-23d9ad240d5c","Type":"ContainerStarted","Data":"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"}
Oct 31 15:18:06.241389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:06.240928    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:06.355825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:06.355759    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" podStartSLOduration=2.493953592 podStartE2EDuration="3.355743298s" totalImagesPullingTime="861.789706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:06.353781097 +0000 UTC m=+1927.080062079" watchObservedRunningTime="2026-10-31 15:18:06.355743298 +0000 UTC m=+1927.082024276"
Oct 31 15:18:07.013429 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:18:07.013386    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:18:12.253128 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:12.253053    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:18:37.013726 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:18:37.013688    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:18:41.809272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:41.809238    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:18:41.809755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:41.809448    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" containerID="cri-o://c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb" gracePeriod=30
Oct 31 15:18:44.540976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:44.540925    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:18:44.541589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:44.540997    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:18:47.192419 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.188641    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:18:47.274556 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274523    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.274778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274594    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.274778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274619    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.274778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274679    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.274778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274757    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.275015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274794    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.275015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274828    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.275015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274890    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:18:47.275015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274917    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.275015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.274967    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:18:47.275281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275048    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:18:47.275281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275061    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\") pod \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\" (UID: \"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d\") "
Oct 31 15:18:47.275396 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275345    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:18:47.275817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275789    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:18:47.275989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275966    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.276067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.275998    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.276067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.276015    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.276067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.276029    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.276067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.276059    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.277659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.277627    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:18:47.277779 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.277763    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:18:47.278095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.278068    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "kube-api-access-wbchx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:18:47.278199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.278184    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token" pod "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" (UID: "c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:18:47.377147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.377117    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.377147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.377146    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.377347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.377156    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.377347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.377165    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbchx\" (UniqueName: \"kubernetes.io/projected/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d-kube-api-access-wbchx\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:18:47.593651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.593592    4549 generic.go:356] "Generic (PLEG): container finished" podID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerID="c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb" exitCode=0
Oct 31 15:18:47.593849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.593639    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"
Oct 31 15:18:47.593849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.593711    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" event={"ID":"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d","Type":"ContainerDied","Data":"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"}
Oct 31 15:18:47.593849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.593788    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd" event={"ID":"c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d","Type":"ContainerDied","Data":"3492de0626ac9ed06c48104d4df4f91e4363015da87607324d21765fcb3f64d7"}
Oct 31 15:18:47.594040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.593852    4549 scope.go:122] "RemoveContainer" containerID="c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"
Oct 31 15:18:47.604446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.604421    4549 scope.go:122] "RemoveContainer" containerID="c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"
Oct 31 15:18:47.604768 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:18:47.604748    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb\": container with ID starting with c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb not found: ID does not exist" containerID="c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"
Oct 31 15:18:47.604827 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.604777    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb"} err="failed to get container status \"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb\": rpc error: code = NotFound desc = could not find container \"c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb\": container with ID starting with c227b5908be36a7fa97a6d59d30ce5bb72b9c4f58afd44c6765fac496f347bbb not found: ID does not exist"
Oct 31 15:18:47.758702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.758607    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:18:47.761726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:47.761705    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-sddhd"]
Oct 31 15:18:49.422680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:18:49.422641    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" path="/var/lib/kubelet/pods/c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d/volumes"
Oct 31 15:19:07.011664 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:07.011614    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:19:25.438828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:25.438798    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:19:30.439856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.439805    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" podUID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" containerName="payload-processing" containerID="cri-o://1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4" gracePeriod=25
Oct 31 15:19:30.737686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.737657    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:19:30.846746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.846717    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\" (UniqueName: \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\") pod \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") "
Oct 31 15:19:30.846949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.846792    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\") pod \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\" (UID: \"57af3aff-aef4-4c79-a9e1-40a63f6b806f\") "
Oct 31 15:19:30.847157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.847131    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume" pod "57af3aff-aef4-4c79-a9e1-40a63f6b806f" (UID: "57af3aff-aef4-4c79-a9e1-40a63f6b806f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:19:30.847282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.847269    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af3aff-aef4-4c79-a9e1-40a63f6b806f-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:30.849035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.849015    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll" pod "57af3aff-aef4-4c79-a9e1-40a63f6b806f" (UID: "57af3aff-aef4-4c79-a9e1-40a63f6b806f"). InnerVolumeSpecName "kube-api-access-2rxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:19:30.941182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.941141    4549 generic.go:356] "Generic (PLEG): container finished" podID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" containerID="1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4" exitCode=0
Oct 31 15:19:30.941441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.941192    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"
Oct 31 15:19:30.941441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.941237    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" event={"ID":"57af3aff-aef4-4c79-a9e1-40a63f6b806f","Type":"ContainerDied","Data":"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"}
Oct 31 15:19:30.941441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.941291    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9" event={"ID":"57af3aff-aef4-4c79-a9e1-40a63f6b806f","Type":"ContainerDied","Data":"a8c6d105f92ad42ba81257d7ec5a7e0eb9c2bc1885ccd8f6f18f83ba21e05822"}
Oct 31 15:19:30.941441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.941318    4549 scope.go:122] "RemoveContainer" containerID="1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"
Oct 31 15:19:30.948244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.948215    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rxll\" (UniqueName: \"kubernetes.io/projected/57af3aff-aef4-4c79-a9e1-40a63f6b806f-kube-api-access-2rxll\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:30.952263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.952246    4549 scope.go:122] "RemoveContainer" containerID="1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"
Oct 31 15:19:30.952800 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:30.952561    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4\": container with ID starting with 1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4 not found: ID does not exist" containerID="1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"
Oct 31 15:19:30.952895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:30.952814    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4"} err="failed to get container status \"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4\": rpc error: code = NotFound desc = could not find container \"1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4\": container with ID starting with 1ab1d10ca9cb72796c2e16679ece931fbd4da81561698aa1e70de25fa22cd8e4 not found: ID does not exist"
Oct 31 15:19:31.104858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:31.104828    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:19:31.108055 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:31.108028    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f76cce-6fbc969c99-gvzp9"]
Oct 31 15:19:31.419638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:31.419544    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" path="/var/lib/kubelet/pods/57af3aff-aef4-4c79-a9e1-40a63f6b806f/volumes"
Oct 31 15:19:37.020262 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:37.020217    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:19:45.452160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:45.452083    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:19:45.452564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:45.452323    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" containerID="cri-o://4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a" gracePeriod=30
Oct 31 15:19:46.936546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:46.936510    4549 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:19:46.936990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:46.936616    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:19:50.774545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.774521    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:19:50.799522 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799485    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.799753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799564    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.799753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799667    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.799753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799710    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.799918 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799781    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.799971 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799919    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:50.800023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.799996    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:19:50.800074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800046    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.800074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800048    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:50.800177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800118    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.800311 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800291    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.800388 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800336    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\") pod \"3b3154a4-64c0-497f-b7a6-005ace06c78c\" (UID: \"3b3154a4-64c0-497f-b7a6-005ace06c78c\") "
Oct 31 15:19:50.800791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800721    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:50.800791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.800763    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:50.801319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.801297    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3154a4-64c0-497f-b7a6-005ace06c78c-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.801405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.801325    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.801405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.801341    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.801405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.801355    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.801405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.801369    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.802550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.802521    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:50.802767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.802737    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:19:50.803005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.802984    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:19:50.803005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.802991    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd" pod "3b3154a4-64c0-497f-b7a6-005ace06c78c" (UID: "3b3154a4-64c0-497f-b7a6-005ace06c78c"). InnerVolumeSpecName "kube-api-access-f8mdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:19:50.903085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.903052    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8mdd\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-kube-api-access-f8mdd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.903085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.903081    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.903085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.903090    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:50.903322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:50.903098    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3154a4-64c0-497f-b7a6-005ace06c78c-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:51.097017 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.096984    4549 generic.go:356] "Generic (PLEG): container finished" podID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerID="4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a" exitCode=0
Oct 31 15:19:51.097208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.097030    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"
Oct 31 15:19:51.097208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.097062    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" event={"ID":"3b3154a4-64c0-497f-b7a6-005ace06c78c","Type":"ContainerDied","Data":"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"}
Oct 31 15:19:51.097208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.097112    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l" event={"ID":"3b3154a4-64c0-497f-b7a6-005ace06c78c","Type":"ContainerDied","Data":"f7ec077046268538fbca437854985502a277d65e2d46e61d948fe1bf6bc174fd"}
Oct 31 15:19:51.097208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.097135    4549 scope.go:122] "RemoveContainer" containerID="4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"
Oct 31 15:19:51.108097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.108079    4549 scope.go:122] "RemoveContainer" containerID="4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"
Oct 31 15:19:51.108385 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:51.108365    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a\": container with ID starting with 4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a not found: ID does not exist" containerID="4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"
Oct 31 15:19:51.108443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.108395    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a"} err="failed to get container status \"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a\": rpc error: code = NotFound desc = could not find container \"4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a\": container with ID starting with 4777325f00457e869b756814179bfe53c3bfe5989b7bd855118840c1ae7c6c5a not found: ID does not exist"
Oct 31 15:19:51.261412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.261373    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:19:51.266501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.266473    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-nft6l"]
Oct 31 15:19:51.421197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:51.421116    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" path="/var/lib/kubelet/pods/3b3154a4-64c0-497f-b7a6-005ace06c78c/volumes"
Oct 31 15:19:52.754341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754306    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:19:52.754869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754855    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy"
Oct 31 15:19:52.754916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754870    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy"
Oct 31 15:19:52.754916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754881    4549 cpu_manager.go:414] "containerMap: removing container" podUID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" containerName="payload-processing"
Oct 31 15:19:52.754916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754885    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" containerName="payload-processing"
Oct 31 15:19:52.754916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754905    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy"
Oct 31 15:19:52.754916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.754910    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy"
Oct 31 15:19:52.755072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.755002    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f0e3dc-23de-44ca-a40b-c0e9d3d2f58d" containerName="istio-proxy"
Oct 31 15:19:52.755072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.755012    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3154a4-64c0-497f-b7a6-005ace06c78c" containerName="istio-proxy"
Oct 31 15:19:52.755072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.755021    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="57af3aff-aef4-4c79-a9e1-40a63f6b806f" containerName="payload-processing"
Oct 31 15:19:52.758237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.758221    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.810494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.810431    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b62069-serving-cert\""
Oct 31 15:19:52.826721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.826685    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.826915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.826878    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97k6f\" (UniqueName: \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.827057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.827031    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.827157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.827139    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.863140 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.863107    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:19:52.928202 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.928170    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.928466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.928257    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97k6f\" (UniqueName: \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.928466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.928308    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.928466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.928374    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.928676 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:52.928498    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-serving-cert: secret "maas-api-e2e-praxis-b-b62069-serving-cert" not found
Oct 31 15:19:52.928676 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:52.928557    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls podName:b9cc7dda-7f94-4cf1-965a-df9237328576 nodeName:}" failed. No retries permitted until 2026-10-31 15:19:53.428542008 +0000 UTC m=+2034.154822965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls") pod "maas-api-e2e-praxis-b-b62069-766468697c-pj28m" (UID: "b9cc7dda-7f94-4cf1-965a-df9237328576") : secret "maas-api-e2e-praxis-b-b62069-serving-cert" not found
Oct 31 15:19:52.929211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.929186    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.931434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.931409    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:52.989067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:52.989036    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97k6f\" (UniqueName: \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:53.434332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:53.434300    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:53.437116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:53.437081    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b62069-766468697c-pj28m\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:53.671602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:53.671538    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:54.056067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:54.056035    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:19:54.057014 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:19:54.056989    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cc7dda_7f94_4cf1_965a_df9237328576.slice/crio-51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c WatchSource:0}: Error finding container 51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c: Status 404 returned error can't find the container with id 51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c
Oct 31 15:19:54.058616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:54.058595    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:19:54.123926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:54.123887    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" event={"ID":"b9cc7dda-7f94-4cf1-965a-df9237328576","Type":"ContainerStarted","Data":"51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c"}
Oct 31 15:19:55.132760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:55.132725    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" event={"ID":"b9cc7dda-7f94-4cf1-965a-df9237328576","Type":"ContainerStarted","Data":"dca3c4c72b7e5ec2988ede578ac52f10eb0e2d9008c3296a69b01a544110a967"}
Oct 31 15:19:55.133178 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:55.132793    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:19:55.250356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:55.250256    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" podStartSLOduration=2.405626064 podStartE2EDuration="3.250241484s" totalImagesPullingTime="844.61542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:19:55.246899531 +0000 UTC m=+2035.973180509" watchObservedRunningTime="2026-10-31 15:19:55.250241484 +0000 UTC m=+2035.976522462"
Oct 31 15:19:57.181182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.181145    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:19:57.181707 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.181397    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery" containerID="cri-o://74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a" gracePeriod=30
Oct 31 15:19:57.519891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.519859    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:19:57.591442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591401    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591439    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591555    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591677    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591703    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591735    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.591956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.591764    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\") pod \"d47f7af3-ef42-438f-a48d-8faa76f9b823\" (UID: \"d47f7af3-ef42-438f-a48d-8faa76f9b823\") "
Oct 31 15:19:57.592077 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.592054    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:19:57.592280 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.592259    4549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-ca-configmap\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.594151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594119    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:19:57.594495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594464    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:19:57.594685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594663    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:19:57.594786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594764    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:19:57.594838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594787    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "kube-api-access-xdscw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:19:57.594981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.594952    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig" pod "d47f7af3-ef42-438f-a48d-8faa76f9b823" (UID: "d47f7af3-ef42-438f-a48d-8faa76f9b823"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:19:57.693826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693785    4549 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-kubeconfig\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.693826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693828    4549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-csr-dns-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.694036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693844    4549 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d47f7af3-ef42-438f-a48d-8faa76f9b823-cacerts\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.694036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693859    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdscw\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-kube-api-access-xdscw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.694036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693875    4549 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d47f7af3-ef42-438f-a48d-8faa76f9b823-local-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:57.694036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:57.693888    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d47f7af3-ef42-438f-a48d-8faa76f9b823-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:19:58.172683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.172634    4549 generic.go:356] "Generic (PLEG): container finished" podID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerID="74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a" exitCode=0
Oct 31 15:19:58.173013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.172988    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" event={"ID":"d47f7af3-ef42-438f-a48d-8faa76f9b823","Type":"ContainerDied","Data":"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"}
Oct 31 15:19:58.173187 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.173136    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5" event={"ID":"d47f7af3-ef42-438f-a48d-8faa76f9b823","Type":"ContainerDied","Data":"57ecaf4a4334ee9ed850adee8c1c2def497048e730224b79a6e12d7e811f19d7"}
Oct 31 15:19:58.173328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.173314    4549 scope.go:122] "RemoveContainer" containerID="74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"
Oct 31 15:19:58.173611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.173592    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"
Oct 31 15:19:58.187308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.187200    4549 scope.go:122] "RemoveContainer" containerID="74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"
Oct 31 15:19:58.187554 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:19:58.187523    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a\": container with ID starting with 74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a not found: ID does not exist" containerID="74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"
Oct 31 15:19:58.187629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.187552    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a"} err="failed to get container status \"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a\": rpc error: code = NotFound desc = could not find container \"74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a\": container with ID starting with 74722f7cbab90d19233168b79a8701eded67490e1433c57c1dd3d98cff4cb04a not found: ID does not exist"
Oct 31 15:19:58.338855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.338822    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:19:58.388323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:58.388278    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hlqx5"]
Oct 31 15:19:59.420052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:19:59.420012    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" path="/var/lib/kubelet/pods/d47f7af3-ef42-438f-a48d-8faa76f9b823/volumes"
Oct 31 15:20:01.145227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:01.145198    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:20:02.310485 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:02.310450    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 15:20:02.311067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:02.310725    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy" containerID="cri-o://ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d" gracePeriod=30
Oct 31 15:20:07.012201 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:20:07.012158    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:20:07.584150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.584083    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 15:20:07.668284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668254    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668483 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668403    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668483 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668446    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668500    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668531    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668531    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:20:07.668633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668599    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668643    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668691    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.668845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668784    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\") pod \"3375e0e0-48b4-4232-95af-66e6751482f0\" (UID: \"3375e0e0-48b4-4232-95af-66e6751482f0\") "
Oct 31 15:20:07.669061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669020    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:20:07.669207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.668956    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:20:07.669763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669290    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.669763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669309    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.669763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669324    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.669763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669607    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:20:07.669763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.669718    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:20:07.671101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.671063    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:20:07.671226 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.671206    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:20:07.671295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.671245    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "kube-api-access-t8n6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:20:07.671496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.671470    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token" pod "3375e0e0-48b4-4232-95af-66e6751482f0" (UID: "3375e0e0-48b4-4232-95af-66e6751482f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:20:07.771216 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771181    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8n6j\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-kube-api-access-t8n6j\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.771216 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771219    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.771456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771234    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375e0e0-48b4-4232-95af-66e6751482f0-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.771456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771250    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375e0e0-48b4-4232-95af-66e6751482f0-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.771456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771264    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375e0e0-48b4-4232-95af-66e6751482f0-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:07.771456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:07.771277    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375e0e0-48b4-4232-95af-66e6751482f0-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:08.271720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.271682    4549 generic.go:356] "Generic (PLEG): container finished" podID="3375e0e0-48b4-4232-95af-66e6751482f0" containerID="ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d" exitCode=0
Oct 31 15:20:08.272223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.271742    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"
Oct 31 15:20:08.272223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.271752    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerDied","Data":"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"}
Oct 31 15:20:08.272223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.271808    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg" event={"ID":"3375e0e0-48b4-4232-95af-66e6751482f0","Type":"ContainerDied","Data":"8633b7100da6c4d5250461428cdc41556c7003113adb2702f1984ef0871590f5"}
Oct 31 15:20:08.272223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.271837    4549 scope.go:122] "RemoveContainer" containerID="ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"
Oct 31 15:20:08.286376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.286358    4549 scope.go:122] "RemoveContainer" containerID="85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"
Oct 31 15:20:08.295080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.295062    4549 scope.go:122] "RemoveContainer" containerID="ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"
Oct 31 15:20:08.295337 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:20:08.295320    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d\": container with ID starting with ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d not found: ID does not exist" containerID="ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"
Oct 31 15:20:08.295396 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.295344    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d"} err="failed to get container status \"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d\": rpc error: code = NotFound desc = could not find container \"ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d\": container with ID starting with ad826f0cff606635aa90a421ebffd40d40239cf69075bade191e67eb5848dc1d not found: ID does not exist"
Oct 31 15:20:08.295396 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.295360    4549 scope.go:122] "RemoveContainer" containerID="85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"
Oct 31 15:20:08.295632 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:20:08.295615    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd\": container with ID starting with 85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd not found: ID does not exist" containerID="85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"
Oct 31 15:20:08.295700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.295635    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd"} err="failed to get container status \"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd\": rpc error: code = NotFound desc = could not find container \"85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd\": container with ID starting with 85a25ca33689c2fb11038b0538d2ba5fee773d40b428bb4d44ca2d1c13ccd9dd not found: ID does not exist"
Oct 31 15:20:08.487080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.487045    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 15:20:08.537398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:08.537315    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-x6jcg"]
Oct 31 15:20:09.420389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:09.420348    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" path="/var/lib/kubelet/pods/3375e0e0-48b4-4232-95af-66e6751482f0/volumes"
Oct 31 15:20:14.657699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:14.657663    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:20:14.658117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:14.657848    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" podUID="9b7de416-7113-420c-96c0-23d9ad240d5c" containerName="maas-api" containerID="cri-o://9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213" gracePeriod=30
Oct 31 15:20:15.026555 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.026515    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:20:15.049030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.048990    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\") pod \"9b7de416-7113-420c-96c0-23d9ad240d5c\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") "
Oct 31 15:20:15.049380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.049104    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\") pod \"9b7de416-7113-420c-96c0-23d9ad240d5c\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") "
Oct 31 15:20:15.049380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.049165    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\" (UniqueName: \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\") pod \"9b7de416-7113-420c-96c0-23d9ad240d5c\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") "
Oct 31 15:20:15.049380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.049368    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") pod \"9b7de416-7113-420c-96c0-23d9ad240d5c\" (UID: \"9b7de416-7113-420c-96c0-23d9ad240d5c\") "
Oct 31 15:20:15.049638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.049424    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca" pod "9b7de416-7113-420c-96c0-23d9ad240d5c" (UID: "9b7de416-7113-420c-96c0-23d9ad240d5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:20:15.049836 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.049818    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7de416-7113-420c-96c0-23d9ad240d5c-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:15.052645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.052521    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls" pod "9b7de416-7113-420c-96c0-23d9ad240d5c" (UID: "9b7de416-7113-420c-96c0-23d9ad240d5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:20:15.052645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.052541    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs" pod "9b7de416-7113-420c-96c0-23d9ad240d5c" (UID: "9b7de416-7113-420c-96c0-23d9ad240d5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:20:15.053306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.053275    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5" pod "9b7de416-7113-420c-96c0-23d9ad240d5c" (UID: "9b7de416-7113-420c-96c0-23d9ad240d5c"). InnerVolumeSpecName "kube-api-access-x2bf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:20:15.157480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.157439    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:15.157480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.157483    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bf5\" (UniqueName: \"kubernetes.io/projected/9b7de416-7113-420c-96c0-23d9ad240d5c-kube-api-access-x2bf5\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:15.157746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.157499    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7de416-7113-420c-96c0-23d9ad240d5c-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:20:15.323979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.323942    4549 generic.go:356] "Generic (PLEG): container finished" podID="9b7de416-7113-420c-96c0-23d9ad240d5c" containerID="9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213" exitCode=0
Oct 31 15:20:15.324139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.323991    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"
Oct 31 15:20:15.324139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.323989    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" event={"ID":"9b7de416-7113-420c-96c0-23d9ad240d5c","Type":"ContainerDied","Data":"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"}
Oct 31 15:20:15.324246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.324164    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj" event={"ID":"9b7de416-7113-420c-96c0-23d9ad240d5c","Type":"ContainerDied","Data":"0ed44bfbfce5e82851379d8e5c6a66097a3edaf715698667c506f294a9e024c0"}
Oct 31 15:20:15.324246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.324205    4549 scope.go:122] "RemoveContainer" containerID="9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"
Oct 31 15:20:15.337083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.337054    4549 scope.go:122] "RemoveContainer" containerID="9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"
Oct 31 15:20:15.337958 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:20:15.337558    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213\": container with ID starting with 9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213 not found: ID does not exist" containerID="9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"
Oct 31 15:20:15.337958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.337638    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213"} err="failed to get container status \"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213\": rpc error: code = NotFound desc = could not find container \"9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213\": container with ID starting with 9a28d7a637d43f92fa16c184e455de7c79e73ddb77cecaaacbcf43075e1be213 not found: ID does not exist"
Oct 31 15:20:15.487377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.487345    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:20:15.490223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:15.490197    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9cc557-848bb8dcdc-x5kpj"]
Oct 31 15:20:17.419689 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:17.419644    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b7de416-7113-420c-96c0-23d9ad240d5c" path="/var/lib/kubelet/pods/9b7de416-7113-420c-96c0-23d9ad240d5c/volumes"
Oct 31 15:20:20.335108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335073    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:20:20.335848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335829    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:20:20.335917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335851    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:20:20.335917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335884    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery"
Oct 31 15:20:20.335917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335893    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery"
Oct 31 15:20:20.336025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335917    4549 cpu_manager.go:414] "containerMap: removing container" podUID="9b7de416-7113-420c-96c0-23d9ad240d5c" containerName="maas-api"
Oct 31 15:20:20.336025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.335925    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b7de416-7113-420c-96c0-23d9ad240d5c" containerName="maas-api"
Oct 31 15:20:20.336093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.336041    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:20:20.336093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.336052    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:20:20.336093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.336069    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47f7af3-ef42-438f-a48d-8faa76f9b823" containerName="discovery"
Oct 31 15:20:20.336093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.336083    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b7de416-7113-420c-96c0-23d9ad240d5c" containerName="maas-api"
Oct 31 15:20:20.341232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.341213    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.401591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.399616    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-da9920-openshift-default-dockercfg-nhc2m\""
Oct 31 15:20:20.449085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.449011    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:20:20.450179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450150    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450336 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450241    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450336 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450290    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450348    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450372    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450397    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450437    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9j9\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450498    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.450737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.450532    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564130    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564204    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564302    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564336    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564375    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564439    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9j9\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564492    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564525    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.564793    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.565796    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.567179    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.567633    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.568080    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.568591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.568355    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.599317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.570234    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.599631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.579319    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.629590 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.628995    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.633261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.633230    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9j9\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\") pod \"e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") " pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:20.665062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:20.665024    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:21.132095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.132063    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:20:21.171406 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:20:21.171376    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177f9bb2_6cf0_43eb_ba3a_416fc4a14360.slice/crio-ee8e21a7c3018d6be9522ce1edfe7b13c013c100ca564ee4e1410c194afb4047 WatchSource:0}: Error finding container ee8e21a7c3018d6be9522ce1edfe7b13c013c100ca564ee4e1410c194afb4047: Status 404 returned error can't find the container with id ee8e21a7c3018d6be9522ce1edfe7b13c013c100ca564ee4e1410c194afb4047
Oct 31 15:20:21.174369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.174333    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:20:21.174457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.174408    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:20:21.174457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.174439    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:20:21.381324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.381286    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerStarted","Data":"edda0c2c834ef4a51ce2b4a3551d2ab024204110503e36ced2460e3fe513a51a"}
Oct 31 15:20:21.381799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.381351    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerStarted","Data":"ee8e21a7c3018d6be9522ce1edfe7b13c013c100ca564ee4e1410c194afb4047"}
Oct 31 15:20:21.498003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.497881    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podStartSLOduration=1.497861389 podStartE2EDuration="1.497861389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:20:21.497072367 +0000 UTC m=+2062.223353347" watchObservedRunningTime="2026-10-31 15:20:21.497861389 +0000 UTC m=+2062.224142369"
Oct 31 15:20:21.665482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:21.665453    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:22.671100 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:22.671072    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:23.393606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:23.393561    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:23.394501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:20:23.394481    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:20:37.015038 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:20:37.014997    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:21:02.780800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:02.780771    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:21:02.896128 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:02.896102    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:21:07.012811 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:21:07.012772    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:21:27.894564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:27.894478    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:21:27.895240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:27.894773    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" containerID="cri-o://1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0" gracePeriod=30
Oct 31 15:21:28.701201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:28.700924    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:21:28.701201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:28.700997    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:21:33.163581 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.163501    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:21:33.244652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244613    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.244864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244697    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.244864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244739    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.244864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244776    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.245033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244949    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:21:33.245033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.244958    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.245141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245065    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.245313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245269    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:33.245601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245543    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:33.245708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245563    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.245781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245703    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.245781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.245748    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\") pod \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\" (UID: \"d7c75f5a-8e37-4bc9-838c-8e081b810a4c\") "
Oct 31 15:21:33.246538 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.246300    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.246538 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.246326    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.246538 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.246343    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.246538 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.246457    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:33.246836 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.246587    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:33.247953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.247928    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:21:33.248878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.248854    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:33.248952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.248895    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "kube-api-access-kn8pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:21:33.250631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.250600    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo" pod "d7c75f5a-8e37-4bc9-838c-8e081b810a4c" (UID: "d7c75f5a-8e37-4bc9-838c-8e081b810a4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:21:33.347807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347773    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.347807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347808    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.348076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347827    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.348076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347843    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.348076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347853    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8pg\" (UniqueName: \"kubernetes.io/projected/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-kube-api-access-kn8pg\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.348076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.347863    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c75f5a-8e37-4bc9-838c-8e081b810a4c-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:33.884465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.884428    4549 generic.go:356] "Generic (PLEG): container finished" podID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerID="1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0" exitCode=0
Oct 31 15:21:33.884676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.884470    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"
Oct 31 15:21:33.884676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.884472    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" event={"ID":"d7c75f5a-8e37-4bc9-838c-8e081b810a4c","Type":"ContainerDied","Data":"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"}
Oct 31 15:21:33.884676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.884637    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v" event={"ID":"d7c75f5a-8e37-4bc9-838c-8e081b810a4c","Type":"ContainerDied","Data":"f5e133b096998867e4eb03cfd80c120986a5c2467a6193fa6dbd259ae7fea599"}
Oct 31 15:21:33.884676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.884668    4549 scope.go:122] "RemoveContainer" containerID="1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"
Oct 31 15:21:33.895479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.895459    4549 scope.go:122] "RemoveContainer" containerID="1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"
Oct 31 15:21:33.895837 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:21:33.895814    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0\": container with ID starting with 1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0 not found: ID does not exist" containerID="1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"
Oct 31 15:21:33.895918 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:33.895850    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0"} err="failed to get container status \"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0\": rpc error: code = NotFound desc = could not find container \"1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0\": container with ID starting with 1aebba8de7991480ff94b5dddb4549aae8911cd9f0ed9aae5c43201f1c5925c0 not found: ID does not exist"
Oct 31 15:21:34.049185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:34.049139    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:21:34.098896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:34.098866    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-wqb2v"]
Oct 31 15:21:35.421283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:35.421242    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" path="/var/lib/kubelet/pods/d7c75f5a-8e37-4bc9-838c-8e081b810a4c/volumes"
Oct 31 15:21:37.014164 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:21:37.014120    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:21:48.929662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:48.929625    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:21:48.930176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:48.929906    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy" containerID="cri-o://3eb0a4c70239cab9686538aae42f1506fc974f567782e46b6a48348957b39f70" gracePeriod=30
Oct 31 15:21:54.143632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.140179    4549 generic.go:356] "Generic (PLEG): container finished" podID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerID="3eb0a4c70239cab9686538aae42f1506fc974f567782e46b6a48348957b39f70" exitCode=0
Oct 31 15:21:54.143632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.140302    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerDied","Data":"3eb0a4c70239cab9686538aae42f1506fc974f567782e46b6a48348957b39f70"}
Oct 31 15:21:54.143632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.141473    4549 scope.go:122] "RemoveContainer" containerID="9a297dea371488177f11b74708c03a83e9e8313306037075584cb7b336992f87"
Oct 31 15:21:54.339593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.339544    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:21:54.379406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379369    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379424    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379450    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379485    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379506    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379594    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379654    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379681    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.379908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379724    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:54.379908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.379851    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\") pod \"8fa34287-61c1-4d84-8d77-81cefe01dfde\" (UID: \"8fa34287-61c1-4d84-8d77-81cefe01dfde\") "
Oct 31 15:21:54.380122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380003    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:54.380185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380149    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:54.380345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380281    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:21:54.380457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380386    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:54.380637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380617    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.380702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380644    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.380702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380661    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.380702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.380677    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa34287-61c1-4d84-8d77-81cefe01dfde-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.382030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.382001    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "kube-api-access-lmx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:21:54.382139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.382123    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:21:54.382560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.382541    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:21:54.382649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.382551    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy" pod "8fa34287-61c1-4d84-8d77-81cefe01dfde" (UID: "8fa34287-61c1-4d84-8d77-81cefe01dfde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:21:54.482649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.482255    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmx2k\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-kube-api-access-lmx2k\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.482649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.482290    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.482649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.482314    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.482649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.482336    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:54.482649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:54.482349    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa34287-61c1-4d84-8d77-81cefe01dfde-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:21:55.151543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.151504    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6" event={"ID":"8fa34287-61c1-4d84-8d77-81cefe01dfde","Type":"ContainerDied","Data":"0903a64a7557fd7af991794472eb8754e9594898aa4bc32df5b1b9056d9bbab4"}
Oct 31 15:21:55.151989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.151609    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"
Oct 31 15:21:55.151989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.151661    4549 scope.go:122] "RemoveContainer" containerID="3eb0a4c70239cab9686538aae42f1506fc974f567782e46b6a48348957b39f70"
Oct 31 15:21:55.329699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.329624    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:21:55.332796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.332758    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9ab5df-openshift-default-778ddf8fc-vcpl6"]
Oct 31 15:21:55.431502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:21:55.431416    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" path="/var/lib/kubelet/pods/8fa34287-61c1-4d84-8d77-81cefe01dfde/volumes"
Oct 31 15:22:01.190996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.190903    4549 generic.go:356] "Generic (PLEG): container finished" podID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerID="edda0c2c834ef4a51ce2b4a3551d2ab024204110503e36ced2460e3fe513a51a" exitCode=0
Oct 31 15:22:01.190996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.190980    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerDied","Data":"edda0c2c834ef4a51ce2b4a3551d2ab024204110503e36ced2460e3fe513a51a"}
Oct 31 15:22:01.191453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.191403    4549 scope.go:122] "RemoveContainer" containerID="edda0c2c834ef4a51ce2b4a3551d2ab024204110503e36ced2460e3fe513a51a"
Oct 31 15:22:01.192974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.192937    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:22:01.193067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.193028    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:22:01.193105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:01.193073    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:22:02.225540 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:02.225496    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerStarted","Data":"c069e9946e9c24b2c368739705f36f77cb7be2931dbc3a7afb27c95d1f510cd6"}
Oct 31 15:22:02.665876 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:02.665835    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:22:02.666203 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:02.666181    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 31 15:22:02.666259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:02.666230    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 31 15:22:03.666238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:03.666201    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 31 15:22:03.666668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:03.666255    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 31 15:22:04.666190 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:04.666149    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 31 15:22:04.666382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:04.666221    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 31 15:22:05.665390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:05.665356    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 31 15:22:05.665565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:05.665411    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 31 15:22:06.665663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:06.665630    4549 patch_prober.go:29] interesting pod/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 31 15:22:06.666071 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:06.665694    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 31 15:22:07.035825 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:22:07.035788    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:22:07.670272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:07.670243    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:22:07.670726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:07.670613    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:22:07.671272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:07.671252    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:22:37.011753 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:22:37.011699    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:22:43.223348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.223263    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:22:43.223862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.223516    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" podUID="b9cc7dda-7f94-4cf1-965a-df9237328576" containerName="maas-api" containerID="cri-o://dca3c4c72b7e5ec2988ede578ac52f10eb0e2d9008c3296a69b01a544110a967" gracePeriod=30
Oct 31 15:22:43.503201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.503160    4549 generic.go:356] "Generic (PLEG): container finished" podID="b9cc7dda-7f94-4cf1-965a-df9237328576" containerID="dca3c4c72b7e5ec2988ede578ac52f10eb0e2d9008c3296a69b01a544110a967" exitCode=0
Oct 31 15:22:43.504053 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.503328    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" event={"ID":"b9cc7dda-7f94-4cf1-965a-df9237328576","Type":"ContainerDied","Data":"dca3c4c72b7e5ec2988ede578ac52f10eb0e2d9008c3296a69b01a544110a967"}
Oct 31 15:22:43.504053 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.503385    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m" event={"ID":"b9cc7dda-7f94-4cf1-965a-df9237328576","Type":"ContainerDied","Data":"51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c"}
Oct 31 15:22:43.504053 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.503410    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51d0643ad23174a78d579a00ac071230fff66c552719788a1552c41751c7e17c"
Oct 31 15:22:43.514442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.514413    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:22:43.659793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.659752    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\") pod \"b9cc7dda-7f94-4cf1-965a-df9237328576\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") "
Oct 31 15:22:43.660014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.659866    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\" (UniqueName: \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\") pod \"b9cc7dda-7f94-4cf1-965a-df9237328576\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") "
Oct 31 15:22:43.660014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.659922    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\") pod \"b9cc7dda-7f94-4cf1-965a-df9237328576\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") "
Oct 31 15:22:43.660014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.659975    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") pod \"b9cc7dda-7f94-4cf1-965a-df9237328576\" (UID: \"b9cc7dda-7f94-4cf1-965a-df9237328576\") "
Oct 31 15:22:43.680968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.680905    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca" pod "b9cc7dda-7f94-4cf1-965a-df9237328576" (UID: "b9cc7dda-7f94-4cf1-965a-df9237328576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:22:43.685011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.684975    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f" pod "b9cc7dda-7f94-4cf1-965a-df9237328576" (UID: "b9cc7dda-7f94-4cf1-965a-df9237328576"). InnerVolumeSpecName "kube-api-access-97k6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:22:43.685183 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.685129    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls" pod "b9cc7dda-7f94-4cf1-965a-df9237328576" (UID: "b9cc7dda-7f94-4cf1-965a-df9237328576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:22:43.685183 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.685172    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs" pod "b9cc7dda-7f94-4cf1-965a-df9237328576" (UID: "b9cc7dda-7f94-4cf1-965a-df9237328576"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:22:43.761767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.761687    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97k6f\" (UniqueName: \"kubernetes.io/projected/b9cc7dda-7f94-4cf1-965a-df9237328576-kube-api-access-97k6f\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:22:43.761767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.761716    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:22:43.761767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.761725    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9cc7dda-7f94-4cf1-965a-df9237328576-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:22:43.761767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:43.761733    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cc7dda-7f94-4cf1-965a-df9237328576-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:22:44.510363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.510330    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"
Oct 31 15:22:44.674414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.674383    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:22:44.723672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.723638    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b62069-766468697c-pj28m"]
Oct 31 15:22:44.877632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.877599    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:22:44.878073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878061    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878075    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878089    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878096    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878103    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878108    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:22:44.878118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878118    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878123    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3375e0e0-48b4-4232-95af-66e6751482f0" containerName="istio-proxy"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878136    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b9cc7dda-7f94-4cf1-965a-df9237328576" containerName="maas-api"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878140    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cc7dda-7f94-4cf1-965a-df9237328576" containerName="maas-api"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878226    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878233    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7c75f5a-8e37-4bc9-838c-8e081b810a4c" containerName="istio-proxy"
Oct 31 15:22:44.878301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.878243    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cc7dda-7f94-4cf1-965a-df9237328576" containerName="maas-api"
Oct 31 15:22:44.881661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.881645    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:44.937461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.937424    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b3065a-serving-cert\""
Oct 31 15:22:44.942985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.942957    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:22:44.975009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.974977    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:44.975168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.975064    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fb8\" (UniqueName: \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:44.975272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.975243    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:44.975389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:44.975359    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.076952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.076918    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fb8\" (UniqueName: \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.077149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.077027    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.077149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.077093    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.077149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.077148    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.077334 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:22:45.077289    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-serving-cert: secret "maas-api-e2e-praxis-a-b3065a-serving-cert" not found
Oct 31 15:22:45.077376 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:22:45.077347    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls podName:b5e95425-f2e6-4896-bba6-12a9bc13c3c7 nodeName:}" failed. No retries permitted until 2026-10-31 15:22:45.577328286 +0000 UTC m=+2206.303609259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls") pod "maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" (UID: "b5e95425-f2e6-4896-bba6-12a9bc13c3c7") : secret "maas-api-e2e-praxis-a-b3065a-serving-cert" not found
Oct 31 15:22:45.077888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.077817    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.079696 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.079663    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.138737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.138657    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fb8\" (UniqueName: \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.419654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.419560    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cc7dda-7f94-4cf1-965a-df9237328576" path="/var/lib/kubelet/pods/b9cc7dda-7f94-4cf1-965a-df9237328576/volumes"
Oct 31 15:22:45.583107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.583074    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.585898 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.585871    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:45.795164 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:45.795134    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:46.180185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:46.180155    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:22:46.181437 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:22:46.181413    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e95425_f2e6_4896_bba6_12a9bc13c3c7.slice/crio-c21e644d6f64daa9583ddabedfc5f24d3c19cd7c80a7d5a871231b896b3a00f2 WatchSource:0}: Error finding container c21e644d6f64daa9583ddabedfc5f24d3c19cd7c80a7d5a871231b896b3a00f2: Status 404 returned error can't find the container with id c21e644d6f64daa9583ddabedfc5f24d3c19cd7c80a7d5a871231b896b3a00f2
Oct 31 15:22:46.523288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:46.523260    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" event={"ID":"b5e95425-f2e6-4896-bba6-12a9bc13c3c7","Type":"ContainerStarted","Data":"c21e644d6f64daa9583ddabedfc5f24d3c19cd7c80a7d5a871231b896b3a00f2"}
Oct 31 15:22:47.530933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:47.530900    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" event={"ID":"b5e95425-f2e6-4896-bba6-12a9bc13c3c7","Type":"ContainerStarted","Data":"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"}
Oct 31 15:22:47.531345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:47.531043    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:22:47.646169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:47.646117    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" podStartSLOduration=2.775690967 podStartE2EDuration="3.646100372s" totalImagesPullingTime="870.409405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:22:47.645009815 +0000 UTC m=+2208.371290790" watchObservedRunningTime="2026-10-31 15:22:47.646100372 +0000 UTC m=+2208.372381347"
Oct 31 15:22:53.544414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:22:53.544379    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:23:07.015276 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:23:07.015230    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:23:37.016526 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:23:37.016486    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:24:01.602526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.602489    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:24:01.603074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.603058    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa34287-61c1-4d84-8d77-81cefe01dfde" containerName="istio-proxy"
Oct 31 15:24:01.606406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.606387    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.662968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.662943    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:24:01.708887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.708855    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bd956c\""
Oct 31 15:24:01.735664 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.709260    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bd956c-dockercfg-ks7qg\""
Oct 31 15:24:01.736027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.710853    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.736352 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.736323    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4v9\" (UniqueName: \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.756047 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.756014    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:24:01.762908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.762871    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:01.814677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.814565    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:24:01.840594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.837364    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4v9\" (UniqueName: \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.840594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.837489    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:01.840594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.837523    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdwt\" (UniqueName: \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:01.840594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.837604    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.840594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.838456    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.894187 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.894119    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4v9\" (UniqueName: \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\") pod \"payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.923377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.923343    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:01.938416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.938386    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:01.938559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.938439    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdwt\" (UniqueName: \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:01.939292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:01.939259    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:02.001244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:02.001207    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdwt\" (UniqueName: \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\") pod \"payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:02.080933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:02.080709    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:02.298630 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:02.298597    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:24:02.299905 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:24:02.299875    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830e59dd_ce10_40b9_b32d_9ea634386244.slice/crio-65fb1c4985b922166b6970b15708a15065cfbe3c0e29596e21b4c3aa5140e6eb WatchSource:0}: Error finding container 65fb1c4985b922166b6970b15708a15065cfbe3c0e29596e21b4c3aa5140e6eb: Status 404 returned error can't find the container with id 65fb1c4985b922166b6970b15708a15065cfbe3c0e29596e21b4c3aa5140e6eb
Oct 31 15:24:02.464080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:02.464053    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:24:02.464601 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:24:02.464551    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dc37aa_25ac_46cb_b852_364e9a6a9e3e.slice/crio-b4cf6677de73b140a282ed2847cfa5274c65c766f579d2877143ab74c2dd9c66 WatchSource:0}: Error finding container b4cf6677de73b140a282ed2847cfa5274c65c766f579d2877143ab74c2dd9c66: Status 404 returned error can't find the container with id b4cf6677de73b140a282ed2847cfa5274c65c766f579d2877143ab74c2dd9c66
Oct 31 15:24:03.148666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.148628    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" event={"ID":"830e59dd-ce10-40b9-b32d-9ea634386244","Type":"ContainerStarted","Data":"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"}
Oct 31 15:24:03.149176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.148695    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" event={"ID":"830e59dd-ce10-40b9-b32d-9ea634386244","Type":"ContainerStarted","Data":"65fb1c4985b922166b6970b15708a15065cfbe3c0e29596e21b4c3aa5140e6eb"}
Oct 31 15:24:03.149176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.148724    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:03.150458 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.150436    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" event={"ID":"00dc37aa-25ac-46cb-b852-364e9a6a9e3e","Type":"ContainerStarted","Data":"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"}
Oct 31 15:24:03.150560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.150488    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:03.150560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.150507    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" event={"ID":"00dc37aa-25ac-46cb-b852-364e9a6a9e3e","Type":"ContainerStarted","Data":"b4cf6677de73b140a282ed2847cfa5274c65c766f579d2877143ab74c2dd9c66"}
Oct 31 15:24:03.262132 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.262079    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" podStartSLOduration=2.262064473 podStartE2EDuration="2.262064473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:03.260753246 +0000 UTC m=+2283.987034225" watchObservedRunningTime="2026-10-31 15:24:03.262064473 +0000 UTC m=+2283.988345450"
Oct 31 15:24:03.374727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:03.374673    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" podStartSLOduration=2.374658686 podStartE2EDuration="2.374658686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:03.372474103 +0000 UTC m=+2284.098755096" watchObservedRunningTime="2026-10-31 15:24:03.374658686 +0000 UTC m=+2284.100939664"
Oct 31 15:24:05.616885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.616846    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:24:05.621322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.621298    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.683838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.683804    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.683925    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.683972    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684023    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684143    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzt2\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684234    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684280    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684399    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.684470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.684455    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.726271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.726240    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:24:05.786169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786131    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786183    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786210    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786287    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786322    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786347    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786397    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786422    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzt2\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786466    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786711    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786737    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786802    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.786917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.786836    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.787290 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.787275    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.788944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.788912    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.789104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.789084    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.842922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.842886    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.843079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.843028    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzt2\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-8s8pv\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:05.936129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:05.936043    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:06.363493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:06.363464    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:24:06.364546 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:24:06.364517    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c068373_ff33_411e_b3df_2f40f763af0e.slice/crio-265de6c8ad2bb42f0c3e55a5a52de1b34a1bfa5a85d6dd4ae437769a109545f1 WatchSource:0}: Error finding container 265de6c8ad2bb42f0c3e55a5a52de1b34a1bfa5a85d6dd4ae437769a109545f1: Status 404 returned error can't find the container with id 265de6c8ad2bb42f0c3e55a5a52de1b34a1bfa5a85d6dd4ae437769a109545f1
Oct 31 15:24:06.367122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:06.367090    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:06.367189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:06.367160    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:06.367227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:06.367189    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:07.041405 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:24:07.041363    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:24:07.180633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.180599    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" event={"ID":"1c068373-ff33-411e-b3df-2f40f763af0e","Type":"ContainerStarted","Data":"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"}
Oct 31 15:24:07.180845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.180654    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" event={"ID":"1c068373-ff33-411e-b3df-2f40f763af0e","Type":"ContainerStarted","Data":"265de6c8ad2bb42f0c3e55a5a52de1b34a1bfa5a85d6dd4ae437769a109545f1"}
Oct 31 15:24:07.394862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.394768    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podStartSLOduration=2.394749712 podStartE2EDuration="2.394749712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:07.344959501 +0000 UTC m=+2288.071240481" watchObservedRunningTime="2026-10-31 15:24:07.394749712 +0000 UTC m=+2288.121030690"
Oct 31 15:24:07.936711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.936678    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:07.938167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.938138    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:07.938308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:07.938193    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:08.930733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:08.930696    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:24:08.935483 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:08.935463    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:08.937381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:08.937356    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:08.937494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:08.937397    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:09.045587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045528    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.045789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045652    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.045789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045679    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.045789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045725    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.045789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045765    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.045789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045789    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.046154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045820    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.046154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045839    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.046154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.045861    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpjs\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.138350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.138311    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:24:09.147157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147124    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147168    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147202    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147220    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147241    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpjs\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147268    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147344    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147368    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.147805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.147421    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.148206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.148180    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.148412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.148374    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.148493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.148442    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.148720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.148698    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.149056 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.149035    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.150400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.150374    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.151916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.151893    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.225641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.224614    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.230025 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.229657    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpjs\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.251960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.251934    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:09.726267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.726232    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:24:09.728593 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:24:09.728544    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ee2425_ccf4_42e6_a524_cff0f8a6e763.slice/crio-c2abf2fb195ffe7f214b007550e638e77b5acdbb5b96ac2889d67474619b6979 WatchSource:0}: Error finding container c2abf2fb195ffe7f214b007550e638e77b5acdbb5b96ac2889d67474619b6979: Status 404 returned error can't find the container with id c2abf2fb195ffe7f214b007550e638e77b5acdbb5b96ac2889d67474619b6979
Oct 31 15:24:09.731133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.731089    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:09.731247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.731159    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:09.731247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.731198    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:24:09.937512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.937474    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:09.937961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:09.937535    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:10.232625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.231465    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" event={"ID":"08ee2425-ccf4-42e6-a524-cff0f8a6e763","Type":"ContainerStarted","Data":"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"}
Oct 31 15:24:10.232625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.231516    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" event={"ID":"08ee2425-ccf4-42e6-a524-cff0f8a6e763","Type":"ContainerStarted","Data":"c2abf2fb195ffe7f214b007550e638e77b5acdbb5b96ac2889d67474619b6979"}
Oct 31 15:24:10.252617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.252556    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:10.254043 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.254024    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:10.254136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.254066    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:10.349042 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.348976    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podStartSLOduration=2.348954806 podStartE2EDuration="2.348954806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:10.348696821 +0000 UTC m=+2291.074977800" watchObservedRunningTime="2026-10-31 15:24:10.348954806 +0000 UTC m=+2291.075235791"
Oct 31 15:24:10.941049 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.941011    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:10.941690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:10.941654    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:11.259350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:11.259208    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:11.259350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:11.259328    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:11.940596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:11.940036    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:11.940596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:11.940113    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:12.252725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:12.252632    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:12.252725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:12.252688    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:12.937623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:12.936728    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:12.937623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:12.936789    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:13.256024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:13.255944    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:13.256403 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:13.256050    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:13.937253 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:13.937215    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 15:24:13.937431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:13.937265    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 15:24:14.162085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:14.162039    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:24:14.162436 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:14.162413    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:24:14.252645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:14.252523    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:14.252645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:14.252628    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:15.035387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:15.035351    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:15.037307 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:15.037079    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:15.038738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:15.038713    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:24:15.253170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:15.253133    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:15.253372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:15.253200    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:16.252699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:16.252664    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:24:16.253123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:16.252721    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:24:17.960165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:17.960135    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:17.961628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:17.961606    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:17.962275 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:17.962252    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:24:19.001986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.001950    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:24:19.006784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.006763    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.064009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.063979    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:24:19.085852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.085820    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d9cddd\"/\"openshift-service-ca.crt\""
Oct 31 15:24:19.085852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.085833    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d9cddd\"/\"kube-root-ca.crt\""
Oct 31 15:24:19.092501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.092472    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d9cddd\"/\"praxis-route-d9cddd-kserve-self-signed-certs\""
Oct 31 15:24:19.092672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.092592    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d9cddd\"/\"default-dockercfg-hbw4b\""
Oct 31 15:24:19.132443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132410    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.132648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132455    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.132648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132551    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.132732 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132677    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.132732 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132719    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnb9h\" (UniqueName: \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.132813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.132801    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234520 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234486    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234534    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234606    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234701    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234754    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnb9h\" (UniqueName: \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.234874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.234806    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.235051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.235013    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.235125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.235100    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.235430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.235392    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.237298 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.237269    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.237529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.237513    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.292099 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.292065    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnb9h\" (UniqueName: \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\") pod \"praxis-route-d9cddd-kserve-89fd79685-rfccs\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") " pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.322519 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.322489    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:19.704079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:19.704052    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:24:19.705139 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:24:19.705110    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5c46d16_6e1d_4b95_bf2a_f80fa672a285.slice/crio-6714a495e11d93235a11778d97520c7f9e6dbaf5db234418a492fe04fc7b7f03 WatchSource:0}: Error finding container 6714a495e11d93235a11778d97520c7f9e6dbaf5db234418a492fe04fc7b7f03: Status 404 returned error can't find the container with id 6714a495e11d93235a11778d97520c7f9e6dbaf5db234418a492fe04fc7b7f03
Oct 31 15:24:20.382096 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:20.382059    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" event={"ID":"a5c46d16-6e1d-4b95-bf2a-f80fa672a285","Type":"ContainerStarted","Data":"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"}
Oct 31 15:24:20.382505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:20.382106    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" event={"ID":"a5c46d16-6e1d-4b95-bf2a-f80fa672a285","Type":"ContainerStarted","Data":"6714a495e11d93235a11778d97520c7f9e6dbaf5db234418a492fe04fc7b7f03"}
Oct 31 15:24:20.496062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:20.495999    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podStartSLOduration=2.495978254 podStartE2EDuration="2.495978254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:20.494431359 +0000 UTC m=+2301.220712334" watchObservedRunningTime="2026-10-31 15:24:20.495978254 +0000 UTC m=+2301.222259233"
Oct 31 15:24:21.323151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:21.323120    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:28.853813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:28.853724    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:24:28.885710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:28.885660    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:24:28.944592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:28.943771    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:24:29.283605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.282670    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:24:29.344606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.332684    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:29.589678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.586943    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:29.622405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.622374    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:24:29.686704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.686670    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:24:29.686878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:24:29.686729    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:24:37.013329 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:24:37.013291    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:25:06.082932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.082895    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:25:06.086974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.086954    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.239991    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240035    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240137    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240186    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240297    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rh85\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240324    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240397    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240423    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.240495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.240480    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.284460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.284426    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:25:06.342262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342175    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342240    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342289    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342307    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342341    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342386    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342436    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rh85\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342464    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342749    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342904 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342839    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.342904 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342890    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.343027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.342933    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.343100 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.343083    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.343135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.343099    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.344941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.344922    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.345073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.345057    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.398707 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.398678    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.398895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.398837    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rh85\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-hp2kz\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.401191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.401168    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:06.830896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.830867    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:25:06.831228 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:25:06.831203    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6c9515_1328_4f34_9c3d_3c9aa65e5250.slice/crio-3e473eb34ea121a7f736e616684a793c05c40e617ef035f56d25f6718c0b2916 WatchSource:0}: Error finding container 3e473eb34ea121a7f736e616684a793c05c40e617ef035f56d25f6718c0b2916: Status 404 returned error can't find the container with id 3e473eb34ea121a7f736e616684a793c05c40e617ef035f56d25f6718c0b2916
Oct 31 15:25:06.834083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.834048    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:06.834173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.834117    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:06.834173 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.834144    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:06.908498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.908463    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" event={"ID":"cc6c9515-1328-4f34-9c3d-3c9aa65e5250","Type":"ContainerStarted","Data":"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"}
Oct 31 15:25:06.908699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:06.908528    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" event={"ID":"cc6c9515-1328-4f34-9c3d-3c9aa65e5250","Type":"ContainerStarted","Data":"3e473eb34ea121a7f736e616684a793c05c40e617ef035f56d25f6718c0b2916"}
Oct 31 15:25:07.013084 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:25:07.013047    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:25:07.024383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:07.024334    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podStartSLOduration=1.024320063 podStartE2EDuration="1.024320063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:07.022965872 +0000 UTC m=+2347.749246919" watchObservedRunningTime="2026-10-31 15:25:07.024320063 +0000 UTC m=+2347.750601112"
Oct 31 15:25:07.409261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:07.401725    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:07.409261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:07.403306    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:07.409261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:07.403375    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:08.403594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:08.403313    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:08.403594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:08.403390    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:09.406009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:09.405969    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:09.406680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:09.406646    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:10.402980 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:10.402942    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:10.403279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:10.403251    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:11.043762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.043725    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:25:11.048068 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.048047    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.124593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.124082    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:25:11.151558 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.151520    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d9cddd-openshift-default-dockercfg-7h672\""
Oct 31 15:25:11.235331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235297    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235347    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235504    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235549    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235603    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235784    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235867 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235824    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235927 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235866    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dwc\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.235984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.235921    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337621    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337665    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dwc\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337720    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337828    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.337902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337853    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337912    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337942    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337954    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.337975    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.338220    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.338343    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338691 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.338526    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.338955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.338936    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.340306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.340285    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.341070 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.341050    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.394294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.394266    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.394512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.394488    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dwc\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:11.406618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.404949    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:11.406618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.405009    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:11.662870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:11.662778    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:12.126184 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:25:12.126149    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31d1972_8315_4354_ae41_0997cb5d8231.slice/crio-00ece7800bdfe25db8b09c918983fb48172e44f7687ae580b4e1f4fbfc300748 WatchSource:0}: Error finding container 00ece7800bdfe25db8b09c918983fb48172e44f7687ae580b4e1f4fbfc300748: Status 404 returned error can't find the container with id 00ece7800bdfe25db8b09c918983fb48172e44f7687ae580b4e1f4fbfc300748
Oct 31 15:25:12.126817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.126771    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:25:12.128957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.128921    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:12.129074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.128995    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:12.129074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.129041    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:25:12.405804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.405703    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:12.406114 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.406081    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:12.971415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.971382    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" event={"ID":"e31d1972-8315-4354-ae41-0997cb5d8231","Type":"ContainerStarted","Data":"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"}
Oct 31 15:25:12.971618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:12.971463    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" event={"ID":"e31d1972-8315-4354-ae41-0997cb5d8231","Type":"ContainerStarted","Data":"00ece7800bdfe25db8b09c918983fb48172e44f7687ae580b4e1f4fbfc300748"}
Oct 31 15:25:13.187296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.187238    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podStartSLOduration=2.187219854 podStartE2EDuration="2.187219854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:13.137027538 +0000 UTC m=+2353.863308529" watchObservedRunningTime="2026-10-31 15:25:13.187219854 +0000 UTC m=+2353.913500835"
Oct 31 15:25:13.407594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.406966    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body=
Oct 31 15:25:13.407594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.407044    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused"
Oct 31 15:25:13.663839 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.663743    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:13.665376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.665351    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 15:25:13.665526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:13.665406    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 15:25:14.663524 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:14.663493    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 15:25:14.664005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:14.663982    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 15:25:14.985283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:14.985164    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:14.986918 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:14.986900    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:14.988742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:14.988702    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:25:15.663877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:15.663843    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 15:25:15.664278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:15.663896    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 15:25:16.664238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:16.664197    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 15:25:16.664660 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:16.664253    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 15:25:17.663219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:17.663186    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Oct 31 15:25:17.663409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:17.663235    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Oct 31 15:25:18.869540 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:18.869508    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:18.870518 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:18.870491    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:18.871540 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:18.871508    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:25:26.840545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:26.840499    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:26.897834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:26.897783    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.067896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.066659    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:25:27.209654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.207115    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.281597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.269458    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.590851 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.581726    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.626799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.626740    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.749756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.749697    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.841965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.841853    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.843989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.842825    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:25:27.855759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.855704    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Oct 31 15:25:27.906407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.906333    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:27.906870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:27.906848    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:25:28.064186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.064152    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 15:25:28.390597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.376917    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:25:28.390597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.340382    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:25:28.390597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.377835    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:25:28.584359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.584299    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 15:25:28.603251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.603221    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:25:28.717708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.716561    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:25:28.809848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:28.809715    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:25:33.132122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:33.132081    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 15:25:33.132621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:33.132328    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy" containerID="cri-o://0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e" gracePeriod=30
Oct 31 15:25:37.012809 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:25:37.012774    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:25:38.400467 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.400444    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 15:25:38.453813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.453728    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.453813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.453795    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.453859    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.453946    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.453995    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454006    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454072    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454191    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454190    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454248    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454271    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:25:38.454320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454297    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\") pod \"094370f9-3682-419e-b874-ccbd046aed7b\" (UID: \"094370f9-3682-419e-b874-ccbd046aed7b\") "
Oct 31 15:25:38.454631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454444    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:25:38.454860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454842    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.454936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454866    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.454936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454879    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.454936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.454892    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.455070 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.455049    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:25:38.456280 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.456252    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:25:38.456702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.456680    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:25:38.456916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.456896    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:25:38.456916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.456906    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb" pod "094370f9-3682-419e-b874-ccbd046aed7b" (UID: "094370f9-3682-419e-b874-ccbd046aed7b"). InnerVolumeSpecName "kube-api-access-vqtxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:25:38.556587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.556539    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.556587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.556563    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/094370f9-3682-419e-b874-ccbd046aed7b-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.556832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.556600    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtxb\" (UniqueName: \"kubernetes.io/projected/094370f9-3682-419e-b874-ccbd046aed7b-kube-api-access-vqtxb\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.556832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.556612    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/094370f9-3682-419e-b874-ccbd046aed7b-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.556832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.556644    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/094370f9-3682-419e-b874-ccbd046aed7b-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:25:38.688621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.688557    4549 generic.go:356] "Generic (PLEG): container finished" podID="094370f9-3682-419e-b874-ccbd046aed7b" containerID="0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e" exitCode=0
Oct 31 15:25:38.688799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.688631    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"
Oct 31 15:25:38.688799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.688637    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" event={"ID":"094370f9-3682-419e-b874-ccbd046aed7b","Type":"ContainerDied","Data":"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"}
Oct 31 15:25:38.688799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.688698    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw" event={"ID":"094370f9-3682-419e-b874-ccbd046aed7b","Type":"ContainerDied","Data":"8c0fe56dbd631991606ce5b7fd2bf43589e376e16a4c6074c3f7d9ea545c15ea"}
Oct 31 15:25:38.688799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.688719    4549 scope.go:122] "RemoveContainer" containerID="0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"
Oct 31 15:25:38.700377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.700349    4549 scope.go:122] "RemoveContainer" containerID="0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"
Oct 31 15:25:38.700654 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:25:38.700633    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e\": container with ID starting with 0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e not found: ID does not exist" containerID="0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"
Oct 31 15:25:38.700746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.700660    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e"} err="failed to get container status \"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e\": rpc error: code = NotFound desc = could not find container \"0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e\": container with ID starting with 0ec101132c138b1bc0d8b002640136dd34da11f3140b5e316033f6c35c99034e not found: ID does not exist"
Oct 31 15:25:38.864096 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.864057    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 15:25:38.875817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:38.875785    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-9xrfw"]
Oct 31 15:25:39.427327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:39.427290    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094370f9-3682-419e-b874-ccbd046aed7b" path="/var/lib/kubelet/pods/094370f9-3682-419e-b874-ccbd046aed7b/volumes"
Oct 31 15:25:55.540767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.540719    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:55.685526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.684985    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:55.766730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.766688    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:55.816992 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.816896    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 15:25:55.851737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.851697    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:25:55.852083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.852053    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:55.865597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:55.856702    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.073795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.073689    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.074459 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.074052    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.204733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.204687    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.369892 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.369756    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.607611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.607547    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.715052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.713755    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": context deadline exceeded"
Oct 31 15:25:56.715052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.714144    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:25:56.779811 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.779766    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:25:56.780381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.780333    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:25:56.850454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.829639    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 15:25:56.851416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.851396    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:25:56.923514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.921156    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:25:56.963230 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:56.963198    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:25:57.088444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.088396    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:25:57.089373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.089347    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:25:57.398652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.384479    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Oct 31 15:25:57.461606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.451957    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:25:57.480823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.480794    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 15:25:57.497319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.497294    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 15:25:57.498596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:25:57.498550    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:26:02.830793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:02.830765    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:26:02.951768 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:02.951742    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:26:02.980037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:02.979996    4549 scope.go:122] "RemoveContainer" containerID="dca3c4c72b7e5ec2988ede578ac52f10eb0e2d9008c3296a69b01a544110a967"
Oct 31 15:26:07.013219 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:26:07.013181    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:26:21.501200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.500479    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:26:21.581705 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.581656    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:21.817632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.817561    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Oct 31 15:26:21.905793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.905749    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:21.921637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.921592    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:21.968727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:21.968664    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:22.076308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.076206    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:26:22.113758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.113704    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:22.169760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.169705    4549 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:22.203767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.203718    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 31 15:26:22.494837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.494740    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:22.505240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.505198    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:22.520116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.520087    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:26:22.591081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.591025    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:26:22.597372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.597330    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:26:22.927370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.927342    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:26:22.928295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:22.928273    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 15:26:25.450721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.450689    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:26:25.451220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.450913    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy" containerID="cri-o://c069e9946e9c24b2c368739705f36f77cb7be2931dbc3a7afb27c95d1f510cd6" gracePeriod=30
Oct 31 15:26:25.635010 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.634864    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:26:25.635554 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.635507    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" podUID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" containerName="maas-api" containerID="cri-o://eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d" gracePeriod=30
Oct 31 15:26:25.941787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.941759    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:26:25.989546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.989515    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\" (UniqueName: \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\") pod \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") "
Oct 31 15:26:25.989546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.989554    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\") pod \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") "
Oct 31 15:26:25.989815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.989723    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\") pod \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") "
Oct 31 15:26:25.989815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.989768    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") pod \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\" (UID: \"b5e95425-f2e6-4896-bba6-12a9bc13c3c7\") "
Oct 31 15:26:25.990553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.990100    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca" pod "b5e95425-f2e6-4896-bba6-12a9bc13c3c7" (UID: "b5e95425-f2e6-4896-bba6-12a9bc13c3c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:26:25.990553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.990521    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:25.992154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.992117    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls" pod "b5e95425-f2e6-4896-bba6-12a9bc13c3c7" (UID: "b5e95425-f2e6-4896-bba6-12a9bc13c3c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:26:25.992154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.992144    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8" pod "b5e95425-f2e6-4896-bba6-12a9bc13c3c7" (UID: "b5e95425-f2e6-4896-bba6-12a9bc13c3c7"). InnerVolumeSpecName "kube-api-access-f8fb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:26:25.992363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:25.992349    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs" pod "b5e95425-f2e6-4896-bba6-12a9bc13c3c7" (UID: "b5e95425-f2e6-4896-bba6-12a9bc13c3c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:26:26.091901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.091855    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:26.091901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.091892    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8fb8\" (UniqueName: \"kubernetes.io/projected/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-kube-api-access-f8fb8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:26.091901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.091908    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e95425-f2e6-4896-bba6-12a9bc13c3c7-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:26.654596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.653144    4549 generic.go:356] "Generic (PLEG): container finished" podID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" containerID="eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d" exitCode=0
Oct 31 15:26:26.654596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.653336    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" event={"ID":"b5e95425-f2e6-4896-bba6-12a9bc13c3c7","Type":"ContainerDied","Data":"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"}
Oct 31 15:26:26.654596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.653387    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4" event={"ID":"b5e95425-f2e6-4896-bba6-12a9bc13c3c7","Type":"ContainerDied","Data":"c21e644d6f64daa9583ddabedfc5f24d3c19cd7c80a7d5a871231b896b3a00f2"}
Oct 31 15:26:26.654596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.653416    4549 scope.go:122] "RemoveContainer" containerID="eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"
Oct 31 15:26:26.654596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.653586    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"
Oct 31 15:26:26.673088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.673060    4549 scope.go:122] "RemoveContainer" containerID="eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"
Oct 31 15:26:26.673468 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:26:26.673435    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d\": container with ID starting with eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d not found: ID does not exist" containerID="eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"
Oct 31 15:26:26.673596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.673479    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d"} err="failed to get container status \"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d\": rpc error: code = NotFound desc = could not find container \"eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d\": container with ID starting with eb301a0ab72fcafe52a677df0bb8d4ef278b9c7a058c8c13bf64f1a1377dc72d not found: ID does not exist"
Oct 31 15:26:26.819424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.819389    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:26:26.822479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:26.822458    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b3065a-575b78bc4b-ghlh4"]
Oct 31 15:26:27.421978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:27.421945    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" path="/var/lib/kubelet/pods/b5e95425-f2e6-4896-bba6-12a9bc13c3c7/volumes"
Oct 31 15:26:30.683419 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.683382    4549 generic.go:356] "Generic (PLEG): container finished" podID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerID="c069e9946e9c24b2c368739705f36f77cb7be2931dbc3a7afb27c95d1f510cd6" exitCode=0
Oct 31 15:26:30.683954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.683471    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerDied","Data":"c069e9946e9c24b2c368739705f36f77cb7be2931dbc3a7afb27c95d1f510cd6"}
Oct 31 15:26:30.683954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.683553    4549 scope.go:122] "RemoveContainer" containerID="edda0c2c834ef4a51ce2b4a3551d2ab024204110503e36ced2460e3fe513a51a"
Oct 31 15:26:30.733978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.733911    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:26:30.760219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760179    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760403 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760310    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760403 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760377    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760435    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760483    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760524    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760609    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760637    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.760746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760699    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:26:30.760746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.760722    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\") pod \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\" (UID: \"177f9bb2-6cf0-43eb-ba3a-416fc4a14360\") "
Oct 31 15:26:30.761236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.761180    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.761236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.761223    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:26:30.761379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.761319    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:26:30.761442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.761387    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:26:30.761531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.761508    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:26:30.762805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.762780    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:26:30.762900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.762817    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "kube-api-access-ds9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:26:30.763350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.763328    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:26:30.763397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.763364    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy" pod "177f9bb2-6cf0-43eb-ba3a-416fc4a14360" (UID: "177f9bb2-6cf0-43eb-ba3a-416fc4a14360"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:26:30.862109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862082    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862106    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862117    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9j9\" (UniqueName: \"kubernetes.io/projected/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-kube-api-access-ds9j9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862127    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862136    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862147    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862156    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:30.862354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:30.862164    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/177f9bb2-6cf0-43eb-ba3a-416fc4a14360-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:31.691708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:31.691624    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"
Oct 31 15:26:31.691708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:31.691625    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp" event={"ID":"177f9bb2-6cf0-43eb-ba3a-416fc4a14360","Type":"ContainerDied","Data":"ee8e21a7c3018d6be9522ce1edfe7b13c013c100ca564ee4e1410c194afb4047"}
Oct 31 15:26:31.691708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:31.691707    4549 scope.go:122] "RemoveContainer" containerID="c069e9946e9c24b2c368739705f36f77cb7be2931dbc3a7afb27c95d1f510cd6"
Oct 31 15:26:31.906892 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:31.906849    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:26:31.956208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:31.956126    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-da9920-openshift-default-7f58cf496f-lkkmp"]
Oct 31 15:26:33.422060 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:33.422024    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" path="/var/lib/kubelet/pods/177f9bb2-6cf0-43eb-ba3a-416fc4a14360/volumes"
Oct 31 15:26:37.023078 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:26:37.023040    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:26:42.490868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:42.489674    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:42.529721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:42.529672    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:42.956602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:42.943920    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:42.956602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:42.946826    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:43.077599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.070136    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 31 15:26:43.077599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.070749    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 15:26:43.124098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.124049    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Oct 31 15:26:43.156256 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.129854    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:43.205022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.204347    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:43.899604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.890078    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:26:43.946764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.946710    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:43.947488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.947467    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:26:43.948464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.948438    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" probeResult="failure" output="Get \"https://10.134.0.116:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:43.948921 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:43.948907    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:26:44.005597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.004073    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:26:44.018419 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.018376    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:26:44.024606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.024113    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 15:26:44.185610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.184051    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:26:44.185610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.184619    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:26:44.552163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:44.552130    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:26:48.901075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:48.901035    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" podUID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" containerName="payload-processing" containerID="cri-o://eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3" gracePeriod=25
Oct 31 15:26:49.019964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.019922    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" podUID="830e59dd-ce10-40b9-b32d-9ea634386244" containerName="payload-pre-processing" containerID="cri-o://5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8" gracePeriod=25
Oct 31 15:26:49.191301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.191267    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:26:49.243615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.242058    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\") pod \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") "
Oct 31 15:26:49.243615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.242173    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\" (UniqueName: \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\") pod \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\" (UID: \"00dc37aa-25ac-46cb-b852-364e9a6a9e3e\") "
Oct 31 15:26:49.243615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.242511    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume" pod "00dc37aa-25ac-46cb-b852-364e9a6a9e3e" (UID: "00dc37aa-25ac-46cb-b852-364e9a6a9e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:26:49.243615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.242753    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:49.248897 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.248869    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt" pod "00dc37aa-25ac-46cb-b852-364e9a6a9e3e" (UID: "00dc37aa-25ac-46cb-b852-364e9a6a9e3e"). InnerVolumeSpecName "kube-api-access-swdwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:26:49.294898 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.294873    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:26:49.336668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.336635    4549 generic.go:356] "Generic (PLEG): container finished" podID="830e59dd-ce10-40b9-b32d-9ea634386244" containerID="5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8" exitCode=0
Oct 31 15:26:49.336942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.336702    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"
Oct 31 15:26:49.336942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.336805    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" event={"ID":"830e59dd-ce10-40b9-b32d-9ea634386244","Type":"ContainerDied","Data":"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"}
Oct 31 15:26:49.336942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.336855    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w" event={"ID":"830e59dd-ce10-40b9-b32d-9ea634386244","Type":"ContainerDied","Data":"65fb1c4985b922166b6970b15708a15065cfbe3c0e29596e21b4c3aa5140e6eb"}
Oct 31 15:26:49.336942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.336886    4549 scope.go:122] "RemoveContainer" containerID="5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"
Oct 31 15:26:49.338874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.338844    4549 generic.go:356] "Generic (PLEG): container finished" podID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" containerID="eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3" exitCode=0
Oct 31 15:26:49.338997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.338941    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" event={"ID":"00dc37aa-25ac-46cb-b852-364e9a6a9e3e","Type":"ContainerDied","Data":"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"}
Oct 31 15:26:49.338997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.338984    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59" event={"ID":"00dc37aa-25ac-46cb-b852-364e9a6a9e3e","Type":"ContainerDied","Data":"b4cf6677de73b140a282ed2847cfa5274c65c766f579d2877143ab74c2dd9c66"}
Oct 31 15:26:49.338997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.338990    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"
Oct 31 15:26:49.343326 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.343302    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\") pod \"830e59dd-ce10-40b9-b32d-9ea634386244\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") "
Oct 31 15:26:49.343962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.343844    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume" pod "830e59dd-ce10-40b9-b32d-9ea634386244" (UID: "830e59dd-ce10-40b9-b32d-9ea634386244"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:26:49.344174 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.344148    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\" (UniqueName: \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\") pod \"830e59dd-ce10-40b9-b32d-9ea634386244\" (UID: \"830e59dd-ce10-40b9-b32d-9ea634386244\") "
Oct 31 15:26:49.344699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.344676    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/830e59dd-ce10-40b9-b32d-9ea634386244-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:49.344833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.344817    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdwt\" (UniqueName: \"kubernetes.io/projected/00dc37aa-25ac-46cb-b852-364e9a6a9e3e-kube-api-access-swdwt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:49.347004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.346976    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9" pod "830e59dd-ce10-40b9-b32d-9ea634386244" (UID: "830e59dd-ce10-40b9-b32d-9ea634386244"). InnerVolumeSpecName "kube-api-access-9s4v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:26:49.350891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.350871    4549 scope.go:122] "RemoveContainer" containerID="5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"
Oct 31 15:26:49.351199 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:26:49.351179    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8\": container with ID starting with 5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8 not found: ID does not exist" containerID="5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"
Oct 31 15:26:49.351269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.351207    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8"} err="failed to get container status \"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8\": rpc error: code = NotFound desc = could not find container \"5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8\": container with ID starting with 5048140d753960e2d7cb84565e248e5895b0e710c5f39c37dded88648a06bbb8 not found: ID does not exist"
Oct 31 15:26:49.351269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.351226    4549 scope.go:122] "RemoveContainer" containerID="eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"
Oct 31 15:26:49.360819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.360804    4549 scope.go:122] "RemoveContainer" containerID="eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"
Oct 31 15:26:49.361102 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:26:49.361084    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3\": container with ID starting with eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3 not found: ID does not exist" containerID="eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"
Oct 31 15:26:49.361166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.361110    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3"} err="failed to get container status \"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3\": rpc error: code = NotFound desc = could not find container \"eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3\": container with ID starting with eff905cebf0585f2ce715cb47276ff3e94b8480ca39ab8322e8e109efddbb5b3 not found: ID does not exist"
Oct 31 15:26:49.446107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.446082    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4v9\" (UniqueName: \"kubernetes.io/projected/830e59dd-ce10-40b9-b32d-9ea634386244-kube-api-access-9s4v9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:26:49.529142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.529105    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:26:49.532477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.532451    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bd956c-6949db8b87-frz59"]
Oct 31 15:26:49.752446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.752358    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:26:49.755351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:49.755328    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bd956c-7f8957cd8c-h999w"]
Oct 31 15:26:51.099462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:51.099428    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:26:51.421349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:51.421267    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" path="/var/lib/kubelet/pods/00dc37aa-25ac-46cb-b852-364e9a6a9e3e/volumes"
Oct 31 15:26:51.421842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:26:51.421821    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830e59dd-ce10-40b9-b32d-9ea634386244" path="/var/lib/kubelet/pods/830e59dd-ce10-40b9-b32d-9ea634386244/volumes"
Oct 31 15:27:06.121271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.121203    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main" containerID="cri-o://e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6" gracePeriod=45
Oct 31 15:27:06.414551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.414520    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:27:06.462347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.462315    4549 generic.go:356] "Generic (PLEG): container finished" podID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerID="e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6" exitCode=0
Oct 31 15:27:06.462529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.462363    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"
Oct 31 15:27:06.462529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.462409    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" event={"ID":"a5c46d16-6e1d-4b95-bf2a-f80fa672a285","Type":"ContainerDied","Data":"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"}
Oct 31 15:27:06.462529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.462456    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs" event={"ID":"a5c46d16-6e1d-4b95-bf2a-f80fa672a285","Type":"ContainerDied","Data":"6714a495e11d93235a11778d97520c7f9e6dbaf5db234418a492fe04fc7b7f03"}
Oct 31 15:27:06.462529 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.462472    4549 scope.go:122] "RemoveContainer" containerID="e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"
Oct 31 15:27:06.471878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.471861    4549 scope.go:122] "RemoveContainer" containerID="e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"
Oct 31 15:27:06.472150 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:27:06.472131    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6\": container with ID starting with e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6 not found: ID does not exist" containerID="e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"
Oct 31 15:27:06.472234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.472160    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6"} err="failed to get container status \"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6\": rpc error: code = NotFound desc = could not find container \"e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6\": container with ID starting with e56172a7706fbfe02744f534120a5e63360c64063e858f19850336ab8fa548e6 not found: ID does not exist"
Oct 31 15:27:06.575203 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575169    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.575398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575317    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\" (UniqueName: \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.575474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575411    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.575474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575452    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.575692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575449    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:06.575692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575543    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.575692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.575681    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\") pod \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\" (UID: \"a5c46d16-6e1d-4b95-bf2a-f80fa672a285\") "
Oct 31 15:27:06.576758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.576195    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:06.576758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.576282    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.576758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.576438    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:06.578748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.578719    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "kube-api-access-jnb9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:27:06.579510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.579482    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:06.579786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.579758    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs" pod "a5c46d16-6e1d-4b95-bf2a-f80fa672a285" (UID: "a5c46d16-6e1d-4b95-bf2a-f80fa672a285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:27:06.677698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.677623    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.677698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.677650    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.677698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.677660    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnb9h\" (UniqueName: \"kubernetes.io/projected/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-kube-api-access-jnb9h\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.677698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.677672    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.677698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.677680    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5c46d16-6e1d-4b95-bf2a-f80fa672a285-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:06.845547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.845512    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:27:06.846094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846075    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846097    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846112    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" containerName="maas-api"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846120    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" containerName="maas-api"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846151    4549 cpu_manager.go:414] "containerMap: removing container" podUID="830e59dd-ce10-40b9-b32d-9ea634386244" containerName="payload-pre-processing"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846162    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830e59dd-ce10-40b9-b32d-9ea634386244" containerName="payload-pre-processing"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846177    4549 cpu_manager.go:414] "containerMap: removing container" podUID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" containerName="payload-processing"
Oct 31 15:27:06.846201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846184    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" containerName="payload-processing"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846214    4549 cpu_manager.go:414] "containerMap: removing container" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846222    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846233    4549 cpu_manager.go:414] "containerMap: removing container" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846241    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846258    4549 cpu_manager.go:414] "containerMap: removing container" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846265    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846382    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" containerName="main"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846395    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e95425-f2e6-4896-bba6-12a9bc13c3c7" containerName="maas-api"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846407    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="094370f9-3682-419e-b874-ccbd046aed7b" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846417    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846428    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="830e59dd-ce10-40b9-b32d-9ea634386244" containerName="payload-pre-processing"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846435    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="177f9bb2-6cf0-43eb-ba3a-416fc4a14360" containerName="istio-proxy"
Oct 31 15:27:06.846560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.846445    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="00dc37aa-25ac-46cb-b852-364e9a6a9e3e" containerName="payload-processing"
Oct 31 15:27:06.850975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.850954    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:06.890102 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.890074    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:27:06.896612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.896588    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-d9cddd/praxis-route-d9cddd-kserve-89fd79685-rfccs"]
Oct 31 15:27:06.903243 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.903218    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-21111f\""
Oct 31 15:27:06.906657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.906632    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:27:06.938048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:06.936778    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-21111f-dockercfg-b5tt6\""
Oct 31 15:27:07.024118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.022055    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhsn\" (UniqueName: \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.024118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.022153    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.024118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.023390    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:27:07.048623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.028207    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.070597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.069985    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:27:07.083901 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:27:07.083867    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:27:07.123360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.123327    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhsn\" (UniqueName: \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.123793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.123395    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.123793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.123529    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhkp\" (UniqueName: \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.123793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.123747    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.124176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.124158    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.180198 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.180160    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhsn\" (UniqueName: \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\") pod \"payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.225138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.225062    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.225312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.225194    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhkp\" (UniqueName: \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.225842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.225819    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.281899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.281871    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhkp\" (UniqueName: \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\") pod \"payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.380318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.380282    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:07.426480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.426437    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c46d16-6e1d-4b95-bf2a-f80fa672a285" path="/var/lib/kubelet/pods/a5c46d16-6e1d-4b95-bf2a-f80fa672a285/volumes"
Oct 31 15:27:07.464990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.464626    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:07.763414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.763389    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:27:07.764018 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:27:07.763995    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df910b1_555d_4cf9_aad4_e8a7bbeaf1a3.slice/crio-1700b280fe35588a1826ce2bad67f0f626bbba22698063cd86e82db68a706004 WatchSource:0}: Error finding container 1700b280fe35588a1826ce2bad67f0f626bbba22698063cd86e82db68a706004: Status 404 returned error can't find the container with id 1700b280fe35588a1826ce2bad67f0f626bbba22698063cd86e82db68a706004
Oct 31 15:27:07.863598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:07.863553    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:27:07.863843 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:27:07.863821    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d3a5b2_3c24_4145_b539_385e6dc4d847.slice/crio-e6b0ff17b02e6fed14a9b01d719470cda04a93e17bad40207c2e88a7cb3433d8 WatchSource:0}: Error finding container e6b0ff17b02e6fed14a9b01d719470cda04a93e17bad40207c2e88a7cb3433d8: Status 404 returned error can't find the container with id e6b0ff17b02e6fed14a9b01d719470cda04a93e17bad40207c2e88a7cb3433d8
Oct 31 15:27:08.492952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.492910    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" event={"ID":"65d3a5b2-3c24-4145-b539-385e6dc4d847","Type":"ContainerStarted","Data":"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"}
Oct 31 15:27:08.493805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.493006    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" event={"ID":"65d3a5b2-3c24-4145-b539-385e6dc4d847","Type":"ContainerStarted","Data":"e6b0ff17b02e6fed14a9b01d719470cda04a93e17bad40207c2e88a7cb3433d8"}
Oct 31 15:27:08.493805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.493038    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:08.495463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.495422    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" event={"ID":"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3","Type":"ContainerStarted","Data":"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"}
Oct 31 15:27:08.495628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.495473    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" event={"ID":"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3","Type":"ContainerStarted","Data":"1700b280fe35588a1826ce2bad67f0f626bbba22698063cd86e82db68a706004"}
Oct 31 15:27:08.495628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.495547    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:08.608068 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:08.608000    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" podStartSLOduration=2.607982369 podStartE2EDuration="2.607982369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:08.605535422 +0000 UTC m=+2469.331816414" watchObservedRunningTime="2026-10-31 15:27:08.607982369 +0000 UTC m=+2469.334263349"
Oct 31 15:27:14.010653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:14.000097    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:27:14.072595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:14.071684    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded"
Oct 31 15:27:14.687775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:14.687733    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:27:14.687977 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:14.687834    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:27:15.593194 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.593130    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" podStartSLOduration=9.593112047 podStartE2EDuration="9.593112047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:08.718400794 +0000 UTC m=+2469.444681772" watchObservedRunningTime="2026-10-31 15:27:15.593112047 +0000 UTC m=+2476.319393034"
Oct 31 15:27:15.594026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.594007    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:27:15.598030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.598010    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.650050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.650015    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 15:27:15.650563 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.650537    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 15:27:15.650750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.650636    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\""
Oct 31 15:27:15.650848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.650662    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 15:27:15.705648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.705609    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:27:15.706175 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706142    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706284    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706364 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706323    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cphtm\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706426    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706461    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706684 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706651    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.706752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.706705    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808171    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808247    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808308    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808329    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cphtm\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808381    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808407    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.808721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.808497    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.809332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.809306    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.811544 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.811516    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.811700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.811554    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.811798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.811773    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.812072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.812053    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.864734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.864667    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.864987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.864962    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cphtm\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-75qr5\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:15.912759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:15.912719    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:16.347297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.347274    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:27:16.349268 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:27:16.349242    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a77c56_4a4c_4b8f_8d20_82e667eb0509.slice/crio-1fc19ba0f2ab3e8090e24ed291e55383b59d1b76fa076dbf3a8345ef3d4d1788 WatchSource:0}: Error finding container 1fc19ba0f2ab3e8090e24ed291e55383b59d1b76fa076dbf3a8345ef3d4d1788: Status 404 returned error can't find the container with id 1fc19ba0f2ab3e8090e24ed291e55383b59d1b76fa076dbf3a8345ef3d4d1788
Oct 31 15:27:16.351697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.351656    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:27:16.351775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.351725    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:27:16.647391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.647285    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" event={"ID":"a1a77c56-4a4c-4b8f-8d20-82e667eb0509","Type":"ContainerStarted","Data":"0131da189f7977a47d4d17b804dd0a49f53a2aee98d7b9f4d485482f1b5d6035"}
Oct 31 15:27:16.647391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.647347    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" event={"ID":"a1a77c56-4a4c-4b8f-8d20-82e667eb0509","Type":"ContainerStarted","Data":"1fc19ba0f2ab3e8090e24ed291e55383b59d1b76fa076dbf3a8345ef3d4d1788"}
Oct 31 15:27:16.647391 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.647376    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:16.765683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:16.765623    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" podStartSLOduration=1.765608269 podStartE2EDuration="1.765608269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:16.762941566 +0000 UTC m=+2477.489222545" watchObservedRunningTime="2026-10-31 15:27:16.765608269 +0000 UTC m=+2477.491889247"
Oct 31 15:27:17.662909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:17.662875    4549 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-75qr5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:27:17.663415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:17.662929    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:27:19.506289 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:19.506258    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:27:19.506802 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:19.506362    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:27:20.618508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:20.618473    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 15:27:20.618937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:20.618687    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy" containerID="cri-o://c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408" gracePeriod=30
Oct 31 15:27:20.661366 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:20.661335    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:27:25.894232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:25.894163    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 15:27:26.041160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041123    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041194    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041266    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041311    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041358    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041391    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041485    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:26.041786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041523    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041591    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.041786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041762    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:26.041966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041780    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:26.041966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041883    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:26.042074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.041999    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:27:26.042129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042104    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\") pod \"a2005099-85d2-48d0-961c-4688729b9099\" (UID: \"a2005099-85d2-48d0-961c-4688729b9099\") "
Oct 31 15:27:26.042685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042602    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.042685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042629    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.042685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042643    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.042685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042656    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.042685 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.042669    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2005099-85d2-48d0-961c-4688729b9099-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.044090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.044068    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:27:26.044192 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.044081    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:27:26.044354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.044330    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:27:26.044425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.044408    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br" pod "a2005099-85d2-48d0-961c-4688729b9099" (UID: "a2005099-85d2-48d0-961c-4688729b9099"). InnerVolumeSpecName "kube-api-access-l49br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:27:26.144015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.143984    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49br\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-kube-api-access-l49br\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.144015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.144014    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2005099-85d2-48d0-961c-4688729b9099-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.144015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.144023    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2005099-85d2-48d0-961c-4688729b9099-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.144283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.144033    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2005099-85d2-48d0-961c-4688729b9099-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:27:26.724257 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.724219    4549 generic.go:356] "Generic (PLEG): container finished" podID="a2005099-85d2-48d0-961c-4688729b9099" containerID="c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408" exitCode=0
Oct 31 15:27:26.724446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.724273    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"
Oct 31 15:27:26.724446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.724316    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" event={"ID":"a2005099-85d2-48d0-961c-4688729b9099","Type":"ContainerDied","Data":"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"}
Oct 31 15:27:26.724446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.724382    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz" event={"ID":"a2005099-85d2-48d0-961c-4688729b9099","Type":"ContainerDied","Data":"11ad460425e93671c76177ceeffa648dc9f3062c1fa944ae0098bc5c7097d5c8"}
Oct 31 15:27:26.724446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.724413    4549 scope.go:122] "RemoveContainer" containerID="c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"
Oct 31 15:27:26.735945 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.735927    4549 scope.go:122] "RemoveContainer" containerID="c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"
Oct 31 15:27:26.736273 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:27:26.736248    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408\": container with ID starting with c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408 not found: ID does not exist" containerID="c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"
Oct 31 15:27:26.736366 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.736280    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408"} err="failed to get container status \"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408\": rpc error: code = NotFound desc = could not find container \"c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408\": container with ID starting with c142407cfde918653c0e30deb7769d7d16ab3658398a8ec5d0edfc6f74e8d408 not found: ID does not exist"
Oct 31 15:27:26.939133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.939096    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 15:27:26.987821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:26.987738    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-4q2wz"]
Oct 31 15:27:27.420989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:27.420960    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2005099-85d2-48d0-961c-4688729b9099" path="/var/lib/kubelet/pods/a2005099-85d2-48d0-961c-4688729b9099/volumes"
Oct 31 15:27:37.013465 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:27:37.013418    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:27:57.420767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.420736    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:27:57.421235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.421221    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy"
Oct 31 15:27:57.421235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.421231    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy"
Oct 31 15:27:57.421310 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.421306    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2005099-85d2-48d0-961c-4688729b9099" containerName="istio-proxy"
Oct 31 15:27:57.424428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.424412    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466187    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466252    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466368    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466420    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466473    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466619    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466645    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwd9\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466674    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.466789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.466764    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.479449 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.479408    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:27:57.568075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568044    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568076    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwd9\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568253    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568295    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568321    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568393    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568447    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568477    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568503    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568522    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568704    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568912 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568893    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.568961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.568946    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.569201 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.569185    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.570914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.570886    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.571050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.571032    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.624839 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.624812    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.624983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.624862    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwd9\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:57.738148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:57.738054    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:58.164271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:58.164247    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:27:58.166622 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:27:58.166594    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c3a7cf_928f_4830_84d4_0922a8337e89.slice/crio-e429a28f0f75068d6d83a70dc21f6bba38af269f8f7027d843cd5231b63cfacf WatchSource:0}: Error finding container e429a28f0f75068d6d83a70dc21f6bba38af269f8f7027d843cd5231b63cfacf: Status 404 returned error can't find the container with id e429a28f0f75068d6d83a70dc21f6bba38af269f8f7027d843cd5231b63cfacf
Oct 31 15:27:58.169172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:58.169132    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:27:58.169276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:58.169199    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:27:58.169276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:58.169227    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:27:59.041592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.041022    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" event={"ID":"19c3a7cf-928f-4830-84d4-0922a8337e89","Type":"ContainerStarted","Data":"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"}
Oct 31 15:27:59.041592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.041143    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" event={"ID":"19c3a7cf-928f-4830-84d4-0922a8337e89","Type":"ContainerStarted","Data":"e429a28f0f75068d6d83a70dc21f6bba38af269f8f7027d843cd5231b63cfacf"}
Oct 31 15:27:59.260806 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.260757    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podStartSLOduration=2.260740132 podStartE2EDuration="2.260740132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:59.223132059 +0000 UTC m=+2519.949413038" watchObservedRunningTime="2026-10-31 15:27:59.260740132 +0000 UTC m=+2519.987021109"
Oct 31 15:27:59.738737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.738700    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:27:59.740464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.740441    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 15:27:59.740523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:27:59.740491    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 15:28:00.739292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:00.739259    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 15:28:00.739754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:00.739310    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 15:28:01.738482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:01.738436    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 15:28:01.738717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:01.738496    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 15:28:02.738605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:02.738550    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 15:28:02.739037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:02.738621    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 15:28:03.739513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:03.739470    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Oct 31 15:28:03.739989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:03.739538    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Oct 31 15:28:04.744204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:04.744174    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:28:04.744783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:04.744758    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:28:04.745517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:04.745497    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:28:07.016457 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:28:07.016413    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:28:28.209516 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.209481    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:28:28.213560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.213537    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.265611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.265549    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5fccd7-serving-cert\""
Oct 31 15:28:28.299394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.299358    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.299598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.299420    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.299598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.299542    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.299694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.299678    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq7fn\" (UniqueName: \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.319005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.318972    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:28:28.400799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.400757    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.401006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.400826    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.401006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.400867    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq7fn\" (UniqueName: \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.401006 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:28:28.400914    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-serving-cert: secret "maas-api-e2e-praxis-a-5fccd7-serving-cert" not found
Oct 31 15:28:28.401006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.400972    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.401006 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:28:28.400994    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls podName:5c2a3fa2-7156-4725-be48-5a4c1e157add nodeName:}" failed. No retries permitted until 2026-10-31 15:28:28.9009725 +0000 UTC m=+2549.627253457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls") pod "maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" (UID: "5c2a3fa2-7156-4725-be48-5a4c1e157add") : secret "maas-api-e2e-praxis-a-5fccd7-serving-cert" not found
Oct 31 15:28:28.401920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.401884    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.404168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.404144    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.460165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.460063    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq7fn\" (UniqueName: \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.912241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.912202    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:28.914815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:28.914783    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:29.127161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:29.127121    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:29.507679 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:29.507643    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:28:29.509880 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:28:29.509742    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2a3fa2_7156_4725_be48_5a4c1e157add.slice/crio-f97b27fdfc71fbf56af03ab39daca93dac944bf2dd13c915eed29ce462297e38 WatchSource:0}: Error finding container f97b27fdfc71fbf56af03ab39daca93dac944bf2dd13c915eed29ce462297e38: Status 404 returned error can't find the container with id f97b27fdfc71fbf56af03ab39daca93dac944bf2dd13c915eed29ce462297e38
Oct 31 15:28:29.510961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:29.510939    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:28:30.286961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:30.286927    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" event={"ID":"5c2a3fa2-7156-4725-be48-5a4c1e157add","Type":"ContainerStarted","Data":"f97b27fdfc71fbf56af03ab39daca93dac944bf2dd13c915eed29ce462297e38"}
Oct 31 15:28:31.294910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:31.294877    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" event={"ID":"5c2a3fa2-7156-4725-be48-5a4c1e157add","Type":"ContainerStarted","Data":"c32b1c4ae465c197490216d9303c410151b7d22bf8b876512db084b2595db21c"}
Oct 31 15:28:31.295318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:31.294984    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:28:31.411155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:31.411097    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" podStartSLOduration=2.537857567 podStartE2EDuration="3.411077357s" totalImagesPullingTime="873.21979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:28:31.407747036 +0000 UTC m=+2552.134028016" watchObservedRunningTime="2026-10-31 15:28:31.411077357 +0000 UTC m=+2552.137358335"
Oct 31 15:28:37.015327 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:28:37.015272    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:28:37.308653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:28:37.308626    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:29:07.012722 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:29:07.012676    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:29:19.668541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.668505    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:29:19.669108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.668757    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" containerID="cri-o://6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb" gracePeriod=30
Oct 31 15:29:19.674316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.674284    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:29:19.674609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.674527    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy" containerID="cri-o://b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109" gracePeriod=30
Oct 31 15:29:19.747479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.747434    4549 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:29:19.747702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:19.747502    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:29:24.977769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:24.977744    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:29:24.980964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:24.980939    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:29:25.089770 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.089741    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.089951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.089804    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.089951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.089844    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.089951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.089869    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.089951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.089901    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090033    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090132    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090158    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:29:25.090197 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090179    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090214    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090265    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090328    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090386    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090433    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090475    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090503    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090540    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\") pod \"e31d1972-8315-4354-ae41-0997cb5d8231\" (UID: \"e31d1972-8315-4354-ae41-0997cb5d8231\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090560    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090597    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090633    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090665    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\") pod \"19c3a7cf-928f-4830-84d4-0922a8337e89\" (UID: \"19c3a7cf-928f-4830-84d4-0922a8337e89\") "
Oct 31 15:29:25.090712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090689    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090826    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.090878    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091342    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091367    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091392    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091424    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091558    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091608    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3a7cf-928f-4830-84d4-0922a8337e89-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091625    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31d1972-8315-4354-ae41-0997cb5d8231-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.091718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091608    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091813    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.091878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.091832    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.093163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.093138    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:29:25.093712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.093606    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:29:25.094254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.094205    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:29:25.094254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.094217    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:29:25.094432 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.094299    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.094816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.094795    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc" pod "e31d1972-8315-4354-ae41-0997cb5d8231" (UID: "e31d1972-8315-4354-ae41-0997cb5d8231"). InnerVolumeSpecName "kube-api-access-w8dwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:29:25.095014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.094998    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "kube-api-access-slwd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:29:25.095151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.095134    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy" pod "19c3a7cf-928f-4830-84d4-0922a8337e89" (UID: "19c3a7cf-928f-4830-84d4-0922a8337e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:29:25.193076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193039    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193069    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dwc\" (UniqueName: \"kubernetes.io/projected/e31d1972-8315-4354-ae41-0997cb5d8231-kube-api-access-w8dwc\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193082    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwd9\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-kube-api-access-slwd9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193091    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31d1972-8315-4354-ae41-0997cb5d8231-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193100    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193108    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193118    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193128    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193136    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31d1972-8315-4354-ae41-0997cb5d8231-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193144    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193153    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3a7cf-928f-4830-84d4-0922a8337e89-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.193325 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.193161    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3a7cf-928f-4830-84d4-0922a8337e89-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:29:25.691615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.691515    4549 generic.go:356] "Generic (PLEG): container finished" podID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerID="6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb" exitCode=0
Oct 31 15:29:25.691615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.691597    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"
Oct 31 15:29:25.691873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.691610    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" event={"ID":"19c3a7cf-928f-4830-84d4-0922a8337e89","Type":"ContainerDied","Data":"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"}
Oct 31 15:29:25.691873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.691664    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff" event={"ID":"19c3a7cf-928f-4830-84d4-0922a8337e89","Type":"ContainerDied","Data":"e429a28f0f75068d6d83a70dc21f6bba38af269f8f7027d843cd5231b63cfacf"}
Oct 31 15:29:25.691873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.691697    4549 scope.go:122] "RemoveContainer" containerID="6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"
Oct 31 15:29:25.693159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.693135    4549 generic.go:356] "Generic (PLEG): container finished" podID="e31d1972-8315-4354-ae41-0997cb5d8231" containerID="b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109" exitCode=0
Oct 31 15:29:25.693274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.693179    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" event={"ID":"e31d1972-8315-4354-ae41-0997cb5d8231","Type":"ContainerDied","Data":"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"}
Oct 31 15:29:25.693274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.693206    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"
Oct 31 15:29:25.693274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.693218    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m" event={"ID":"e31d1972-8315-4354-ae41-0997cb5d8231","Type":"ContainerDied","Data":"00ece7800bdfe25db8b09c918983fb48172e44f7687ae580b4e1f4fbfc300748"}
Oct 31 15:29:25.715870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.715848    4549 scope.go:122] "RemoveContainer" containerID="6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"
Oct 31 15:29:25.716245 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:29:25.716218    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb\": container with ID starting with 6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb not found: ID does not exist" containerID="6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"
Oct 31 15:29:25.716394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.716258    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb"} err="failed to get container status \"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb\": rpc error: code = NotFound desc = could not find container \"6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb\": container with ID starting with 6c4ab05f89493afc16e92b882872680049a63bf4ca0d9b7fe155f45bba3b2fdb not found: ID does not exist"
Oct 31 15:29:25.716394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.716282    4549 scope.go:122] "RemoveContainer" containerID="b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"
Oct 31 15:29:25.726079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.726058    4549 scope.go:122] "RemoveContainer" containerID="b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"
Oct 31 15:29:25.726424 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:29:25.726403    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109\": container with ID starting with b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109 not found: ID does not exist" containerID="b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"
Oct 31 15:29:25.726496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.726437    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109"} err="failed to get container status \"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109\": rpc error: code = NotFound desc = could not find container \"b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109\": container with ID starting with b743dfcf814a60ef4eadc78d9476ce145edd9479eed705eede041f65147ec109 not found: ID does not exist"
Oct 31 15:29:25.855653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.855624    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:29:25.906739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:25.906702    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hvw6m"]
Oct 31 15:29:26.012902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:26.012811    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:29:26.021968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:26.021939    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-2xcff"]
Oct 31 15:29:27.420928 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:27.420896    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" path="/var/lib/kubelet/pods/19c3a7cf-928f-4830-84d4-0922a8337e89/volumes"
Oct 31 15:29:27.421558 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:29:27.421536    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" path="/var/lib/kubelet/pods/e31d1972-8315-4354-ae41-0997cb5d8231/volumes"
Oct 31 15:29:37.013800 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:29:37.013758    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:30:00.186385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.186351    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:00.187106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187005    4549 cpu_manager.go:414] "containerMap: removing container" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy"
Oct 31 15:30:00.187106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187025    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy"
Oct 31 15:30:00.187106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187046    4549 cpu_manager.go:414] "containerMap: removing container" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy"
Oct 31 15:30:00.187106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187051    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy"
Oct 31 15:30:00.187331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187177    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31d1972-8315-4354-ae41-0997cb5d8231" containerName="istio-proxy"
Oct 31 15:30:00.187331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.187189    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c3a7cf-928f-4830-84d4-0922a8337e89" containerName="istio-proxy"
Oct 31 15:30:00.190661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.190641    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.210709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.210673    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:00.215357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.215330    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:00.215502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.215445    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.219456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.219435    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.238318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.238288    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:00.242281 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.242258    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.271633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.271598    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.271821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.271663    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf97\" (UniqueName: \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.271821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.271785    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.271958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.271939    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.272077 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.272023    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgz5m\" (UniqueName: \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.272123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.272072    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hqk\" (UniqueName: \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.320968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.320020    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:00.323439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.323409    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 15:30:00.357763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.357726    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:00.366606 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.361631    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386312    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386380    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgqp\" (UniqueName: \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386438    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgz5m\" (UniqueName: \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386468    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hqk\" (UniqueName: \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386534    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386627    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386660    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf97\" (UniqueName: \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.386729    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.387217    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.387519    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.389015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.388364    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.392671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.392390    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:00.444551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.444465    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgz5m\" (UniqueName: \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\") pod \"maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.444796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.444770    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf97\" (UniqueName: \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.444865 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.444795    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hqk\" (UniqueName: \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\") pod \"maas-api-key-cleanup-29891010-dszrn\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.487461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.487424    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgqp\" (UniqueName: \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.487694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.487536    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.488181 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.488161    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.505765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.505729    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:00.533783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.533744    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:00.543785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.543553    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:00.544680 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.544649    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgqp\" (UniqueName: \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.559764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.559733    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:00.899854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.899818    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:00.900709 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:30:00.900681    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceb4c53_3254_4103_8fb8_d026078649d1.slice/crio-a9619418314514df6d6e1badab1b8a7b07dd88c504032e86f9b851030b6892f8 WatchSource:0}: Error finding container a9619418314514df6d6e1badab1b8a7b07dd88c504032e86f9b851030b6892f8: Status 404 returned error can't find the container with id a9619418314514df6d6e1badab1b8a7b07dd88c504032e86f9b851030b6892f8
Oct 31 15:30:00.947278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.947238    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerStarted","Data":"a9619418314514df6d6e1badab1b8a7b07dd88c504032e86f9b851030b6892f8"}
Oct 31 15:30:00.949279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:00.949254    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:00.952929 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:30:00.952897    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf54e62a_f0b8_4fc0_a302_f9a30388f012.slice/crio-cd305a42ac4796fa9c0a9d1d4cd7fc8611acde32041cbeebbd51b83bcc625876 WatchSource:0}: Error finding container cd305a42ac4796fa9c0a9d1d4cd7fc8611acde32041cbeebbd51b83bcc625876: Status 404 returned error can't find the container with id cd305a42ac4796fa9c0a9d1d4cd7fc8611acde32041cbeebbd51b83bcc625876
Oct 31 15:30:01.001536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.001432    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:01.008815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.008462    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:01.101658 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:30:01.101622    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ae9101_f2e9_40c8_9043_a35beb67562b.slice/crio-0d86d131feb47b1fdf2c1d70712ca11c18ac52640ce294f6a61d3ea616f5295b WatchSource:0}: Error finding container 0d86d131feb47b1fdf2c1d70712ca11c18ac52640ce294f6a61d3ea616f5295b: Status 404 returned error can't find the container with id 0d86d131feb47b1fdf2c1d70712ca11c18ac52640ce294f6a61d3ea616f5295b
Oct 31 15:30:01.963739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.963699    4549 generic.go:356] "Generic (PLEG): container finished" podID="fceb4c53-3254-4103-8fb8-d026078649d1" containerID="7c5da6f11f91ba5d36bf852156504b63976cbb58e0a3b367440ad967e658d93d" exitCode=60
Oct 31 15:30:01.964252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.963782    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerDied","Data":"7c5da6f11f91ba5d36bf852156504b63976cbb58e0a3b367440ad967e658d93d"}
Oct 31 15:30:01.964252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.964126    4549 scope.go:122] "RemoveContainer" containerID="7c5da6f11f91ba5d36bf852156504b63976cbb58e0a3b367440ad967e658d93d"
Oct 31 15:30:01.966922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.966172    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerStarted","Data":"e0f64013c8441bbfd16d8c7fc0eb9b27796e1359db7a6c29799d60ca876000d8"}
Oct 31 15:30:01.966922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.966228    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerStarted","Data":"f7bf44539d2bc6aee0660a4ab152c8bf56acce8b2a259723d0d8a7378b1acf4d"}
Oct 31 15:30:01.968889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.968866    4549 generic.go:356] "Generic (PLEG): container finished" podID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerID="fa20286026434b418656558dbbad54bcfd8a95599de9a69fca2b413733d989bf" exitCode=60
Oct 31 15:30:01.968995 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.968953    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerDied","Data":"fa20286026434b418656558dbbad54bcfd8a95599de9a69fca2b413733d989bf"}
Oct 31 15:30:01.969062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.969011    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerStarted","Data":"cd305a42ac4796fa9c0a9d1d4cd7fc8611acde32041cbeebbd51b83bcc625876"}
Oct 31 15:30:01.969250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.969231    4549 scope.go:122] "RemoveContainer" containerID="fa20286026434b418656558dbbad54bcfd8a95599de9a69fca2b413733d989bf"
Oct 31 15:30:01.978172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.978016    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerStarted","Data":"dce09f5cb72dcc5130066665be1317731c393148a3030c60bc69d71a706f1de6"}
Oct 31 15:30:01.978172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:01.978073    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerStarted","Data":"0d86d131feb47b1fdf2c1d70712ca11c18ac52640ce294f6a61d3ea616f5295b"}
Oct 31 15:30:02.192862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.192760    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" podStartSLOduration=2.192741687 podStartE2EDuration="2.192741687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.188845811 +0000 UTC m=+2642.915126793" watchObservedRunningTime="2026-10-31 15:30:02.192741687 +0000 UTC m=+2642.919022664"
Oct 31 15:30:02.415011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.414950    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" podStartSLOduration=2.414932363 podStartE2EDuration="2.414932363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:02.413306686 +0000 UTC m=+2643.139587666" watchObservedRunningTime="2026-10-31 15:30:02.414932363 +0000 UTC m=+2643.141213342"
Oct 31 15:30:02.986843 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.986753    4549 generic.go:356] "Generic (PLEG): container finished" podID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerID="863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2" exitCode=60
Oct 31 15:30:02.987308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.986857    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerDied","Data":"863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2"}
Oct 31 15:30:02.987308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.986916    4549 scope.go:122] "RemoveContainer" containerID="fa20286026434b418656558dbbad54bcfd8a95599de9a69fca2b413733d989bf"
Oct 31 15:30:02.987308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.987243    4549 scope.go:122] "RemoveContainer" containerID="863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2"
Oct 31 15:30:02.987619 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:02.987593    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg_odh-ai-gateway-infra(af54e62a-f0b8-4fc0-a302-f9a30388f012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012"
Oct 31 15:30:02.988835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.988811    4549 generic.go:356] "Generic (PLEG): container finished" podID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerID="dce09f5cb72dcc5130066665be1317731c393148a3030c60bc69d71a706f1de6" exitCode=60
Oct 31 15:30:02.988932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.988919    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerDied","Data":"dce09f5cb72dcc5130066665be1317731c393148a3030c60bc69d71a706f1de6"}
Oct 31 15:30:02.989144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.989130    4549 scope.go:122] "RemoveContainer" containerID="dce09f5cb72dcc5130066665be1317731c393148a3030c60bc69d71a706f1de6"
Oct 31 15:30:02.991242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.991216    4549 generic.go:356] "Generic (PLEG): container finished" podID="fceb4c53-3254-4103-8fb8-d026078649d1" containerID="ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1" exitCode=60
Oct 31 15:30:02.991345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.991292    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerDied","Data":"ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1"}
Oct 31 15:30:02.991558 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.991539    4549 scope.go:122] "RemoveContainer" containerID="ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1"
Oct 31 15:30:02.992252 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:02.992228    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf_odh-ai-gateway-infra(fceb4c53-3254-4103-8fb8-d026078649d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" podUID="fceb4c53-3254-4103-8fb8-d026078649d1"
Oct 31 15:30:02.993397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.993376    4549 generic.go:356] "Generic (PLEG): container finished" podID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerID="e0f64013c8441bbfd16d8c7fc0eb9b27796e1359db7a6c29799d60ca876000d8" exitCode=60
Oct 31 15:30:02.993492 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.993424    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerDied","Data":"e0f64013c8441bbfd16d8c7fc0eb9b27796e1359db7a6c29799d60ca876000d8"}
Oct 31 15:30:02.993764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:02.993746    4549 scope.go:122] "RemoveContainer" containerID="e0f64013c8441bbfd16d8c7fc0eb9b27796e1359db7a6c29799d60ca876000d8"
Oct 31 15:30:03.003498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:03.003477    4549 scope.go:122] "RemoveContainer" containerID="7c5da6f11f91ba5d36bf852156504b63976cbb58e0a3b367440ad967e658d93d"
Oct 31 15:30:04.002542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.002511    4549 generic.go:356] "Generic (PLEG): container finished" podID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerID="b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63" exitCode=60
Oct 31 15:30:04.003030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.002607    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerDied","Data":"b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63"}
Oct 31 15:30:04.003030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.002677    4549 scope.go:122] "RemoveContainer" containerID="e0f64013c8441bbfd16d8c7fc0eb9b27796e1359db7a6c29799d60ca876000d8"
Oct 31 15:30:04.003030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.002965    4549 scope.go:122] "RemoveContainer" containerID="b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63"
Oct 31 15:30:04.003288 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:04.003263    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891010-dszrn_odh-ai-gateway-infra(19b18dea-261b-4335-bf0e-8a9a7e2d9fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf"
Oct 31 15:30:04.004690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.004673    4549 scope.go:122] "RemoveContainer" containerID="863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2"
Oct 31 15:30:04.004991 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:04.004972    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg_odh-ai-gateway-infra(af54e62a-f0b8-4fc0-a302-f9a30388f012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012"
Oct 31 15:30:04.006217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.006198    4549 generic.go:356] "Generic (PLEG): container finished" podID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerID="2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21" exitCode=60
Oct 31 15:30:04.006296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.006273    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerDied","Data":"2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21"}
Oct 31 15:30:04.006621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.006564    4549 scope.go:122] "RemoveContainer" containerID="2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21"
Oct 31 15:30:04.006964 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:04.006867    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp_odh-ai-gateway-infra(27ae9101-f2e9-40c8-9043-a35beb67562b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b"
Oct 31 15:30:04.009292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.008395    4549 scope.go:122] "RemoveContainer" containerID="ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1"
Oct 31 15:30:04.009292 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:04.008679    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf_odh-ai-gateway-infra(fceb4c53-3254-4103-8fb8-d026078649d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" podUID="fceb4c53-3254-4103-8fb8-d026078649d1"
Oct 31 15:30:04.051412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:04.051390    4549 scope.go:122] "RemoveContainer" containerID="dce09f5cb72dcc5130066665be1317731c393148a3030c60bc69d71a706f1de6"
Oct 31 15:30:05.020869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:05.020836    4549 scope.go:122] "RemoveContainer" containerID="2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21"
Oct 31 15:30:05.021402 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:05.021201    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp_odh-ai-gateway-infra(27ae9101-f2e9-40c8-9043-a35beb67562b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b"
Oct 31 15:30:05.023885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:05.023848    4549 scope.go:122] "RemoveContainer" containerID="b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63"
Oct 31 15:30:05.024167 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:05.024147    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891010-dszrn_odh-ai-gateway-infra(19b18dea-261b-4335-bf0e-8a9a7e2d9fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf"
Oct 31 15:30:07.021972 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:07.021922    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:30:15.413126 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:15.413035    4549 scope.go:122] "RemoveContainer" containerID="863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2"
Oct 31 15:30:15.413640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:15.413165    4549 scope.go:122] "RemoveContainer" containerID="b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63"
Oct 31 15:30:15.413640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:15.413248    4549 scope.go:122] "RemoveContainer" containerID="2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21"
Oct 31 15:30:16.109102 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:16.109071    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerStarted","Data":"ca66854213e7de043fc79ef8fa4856ae67160b1cacabb6efd53c3bdcf9a21a4b"}
Oct 31 15:30:16.111155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:16.111129    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerStarted","Data":"119854b0f110cf8baea10f0dba8870c725e46ea72caf92b69280bb398ef0f0d6"}
Oct 31 15:30:16.113159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:16.113136    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerStarted","Data":"e1963ef41bd7ceb4dd15d710b930c6243d1e2bb85c81a3da5b0b70f17538994e"}
Oct 31 15:30:16.224224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:16.224169    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" podStartSLOduration=16.224154963 podStartE2EDuration="16.224154963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:16.220730243 +0000 UTC m=+2656.947011223" watchObservedRunningTime="2026-10-31 15:30:16.224154963 +0000 UTC m=+2656.950435940"
Oct 31 15:30:16.412960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:16.412933    4549 scope.go:122] "RemoveContainer" containerID="ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1"
Oct 31 15:30:17.122176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.122143    4549 generic.go:356] "Generic (PLEG): container finished" podID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerID="e1963ef41bd7ceb4dd15d710b930c6243d1e2bb85c81a3da5b0b70f17538994e" exitCode=60
Oct 31 15:30:17.122731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.122227    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerDied","Data":"e1963ef41bd7ceb4dd15d710b930c6243d1e2bb85c81a3da5b0b70f17538994e"}
Oct 31 15:30:17.122731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.122336    4549 scope.go:122] "RemoveContainer" containerID="b0d1500f1f7e95ad172b0a9e1af30c96b069b4cde298410c461229a1a65c6a63"
Oct 31 15:30:17.122862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.122757    4549 scope.go:122] "RemoveContainer" containerID="e1963ef41bd7ceb4dd15d710b930c6243d1e2bb85c81a3da5b0b70f17538994e"
Oct 31 15:30:17.123089 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:17.123052    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891010-dszrn_odh-ai-gateway-infra(19b18dea-261b-4335-bf0e-8a9a7e2d9fdf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf"
Oct 31 15:30:17.124369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.124345    4549 generic.go:356] "Generic (PLEG): container finished" podID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerID="ca66854213e7de043fc79ef8fa4856ae67160b1cacabb6efd53c3bdcf9a21a4b" exitCode=60
Oct 31 15:30:17.124475 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.124413    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerDied","Data":"ca66854213e7de043fc79ef8fa4856ae67160b1cacabb6efd53c3bdcf9a21a4b"}
Oct 31 15:30:17.124828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.124813    4549 scope.go:122] "RemoveContainer" containerID="ca66854213e7de043fc79ef8fa4856ae67160b1cacabb6efd53c3bdcf9a21a4b"
Oct 31 15:30:17.125026 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:17.125007    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg_odh-ai-gateway-infra(af54e62a-f0b8-4fc0-a302-f9a30388f012)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012"
Oct 31 15:30:17.126400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.126379    4549 generic.go:356] "Generic (PLEG): container finished" podID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerID="119854b0f110cf8baea10f0dba8870c725e46ea72caf92b69280bb398ef0f0d6" exitCode=60
Oct 31 15:30:17.126472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.126405    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerDied","Data":"119854b0f110cf8baea10f0dba8870c725e46ea72caf92b69280bb398ef0f0d6"}
Oct 31 15:30:17.126717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.126703    4549 scope.go:122] "RemoveContainer" containerID="119854b0f110cf8baea10f0dba8870c725e46ea72caf92b69280bb398ef0f0d6"
Oct 31 15:30:17.126940 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:17.126918    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp_odh-ai-gateway-infra(27ae9101-f2e9-40c8-9043-a35beb67562b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b"
Oct 31 15:30:17.128656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.128637    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerStarted","Data":"9472d0dd5343700007d0014cecb06713b079e0df2b966c1f038c029e3740a139"}
Oct 31 15:30:17.136626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.136600    4549 scope.go:122] "RemoveContainer" containerID="863ee7813ca6ab8ccf324b0e8a78e648fcc96557f9c45a06fd6dc000b017cfb2"
Oct 31 15:30:17.150040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.150016    4549 scope.go:122] "RemoveContainer" containerID="2bbbee99243877af44ca8ca318f92f9769059c2e727b9a9be09da3f5e0cfbc21"
Oct 31 15:30:17.232331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.232301    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:17.349997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.349969    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:17.351848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.351791    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" podStartSLOduration=17.351770975 podStartE2EDuration="17.351770975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:17.347786448 +0000 UTC m=+2658.074067427" watchObservedRunningTime="2026-10-31 15:30:17.351770975 +0000 UTC m=+2658.078051955"
Oct 31 15:30:17.456599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:17.456551    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:18.143214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.143181    4549 generic.go:356] "Generic (PLEG): container finished" podID="fceb4c53-3254-4103-8fb8-d026078649d1" containerID="9472d0dd5343700007d0014cecb06713b079e0df2b966c1f038c029e3740a139" exitCode=60
Oct 31 15:30:18.143716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.143243    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerDied","Data":"9472d0dd5343700007d0014cecb06713b079e0df2b966c1f038c029e3740a139"}
Oct 31 15:30:18.143716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.143297    4549 scope.go:122] "RemoveContainer" containerID="ba0f08edca2f64f6dd4443b3cf4b793eb5b474f63fe142c606b0e02f410d47b1"
Oct 31 15:30:18.143830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.143715    4549 scope.go:122] "RemoveContainer" containerID="9472d0dd5343700007d0014cecb06713b079e0df2b966c1f038c029e3740a139"
Oct 31 15:30:18.144091 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:18.144057    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf_odh-ai-gateway-infra(fceb4c53-3254-4103-8fb8-d026078649d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" podUID="fceb4c53-3254-4103-8fb8-d026078649d1"
Oct 31 15:30:18.324956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.324929    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:18.360597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.360550    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:18.362940 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.362916    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:18.384251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.384216    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:18.440505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440426    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\") pod \"af54e62a-f0b8-4fc0-a302-f9a30388f012\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") "
Oct 31 15:30:18.440505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440496    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\" (UniqueName: \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\") pod \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") "
Oct 31 15:30:18.440773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440556    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\" (UniqueName: \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\") pod \"af54e62a-f0b8-4fc0-a302-f9a30388f012\" (UID: \"af54e62a-f0b8-4fc0-a302-f9a30388f012\") "
Oct 31 15:30:18.440773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440680    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\") pod \"27ae9101-f2e9-40c8-9043-a35beb67562b\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") "
Oct 31 15:30:18.440773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440759    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\" (UniqueName: \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\") pod \"27ae9101-f2e9-40c8-9043-a35beb67562b\" (UID: \"27ae9101-f2e9-40c8-9043-a35beb67562b\") "
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440793    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\") pod \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\" (UID: \"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf\") "
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.440993    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca" pod "af54e62a-f0b8-4fc0-a302-f9a30388f012" (UID: "af54e62a-f0b8-4fc0-a302-f9a30388f012"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.441035    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca" pod "27ae9101-f2e9-40c8-9043-a35beb67562b" (UID: "27ae9101-f2e9-40c8-9043-a35beb67562b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.441290    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca" pod "19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" (UID: "19b18dea-261b-4335-bf0e-8a9a7e2d9fdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.441388    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.441405    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af54e62a-f0b8-4fc0-a302-f9a30388f012-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:18.441497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.441416    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27ae9101-f2e9-40c8-9043-a35beb67562b-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:18.443043 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.443021    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk" pod "19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" (UID: "19b18dea-261b-4335-bf0e-8a9a7e2d9fdf"). InnerVolumeSpecName "kube-api-access-g7hqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:30:18.443115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.443042    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97" pod "af54e62a-f0b8-4fc0-a302-f9a30388f012" (UID: "af54e62a-f0b8-4fc0-a302-f9a30388f012"). InnerVolumeSpecName "kube-api-access-gdf97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:30:18.443174 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.443134    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp" pod "27ae9101-f2e9-40c8-9043-a35beb67562b" (UID: "27ae9101-f2e9-40c8-9043-a35beb67562b"). InnerVolumeSpecName "kube-api-access-nqgqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:30:18.542282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.542249    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7hqk\" (UniqueName: \"kubernetes.io/projected/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf-kube-api-access-g7hqk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:18.542282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.542279    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdf97\" (UniqueName: \"kubernetes.io/projected/af54e62a-f0b8-4fc0-a302-f9a30388f012-kube-api-access-gdf97\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:18.542282 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:18.542288    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgqp\" (UniqueName: \"kubernetes.io/projected/27ae9101-f2e9-40c8-9043-a35beb67562b-kube-api-access-nqgqp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:19.153865 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.153833    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"
Oct 31 15:30:19.154349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.153879    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn" event={"ID":"19b18dea-261b-4335-bf0e-8a9a7e2d9fdf","Type":"ContainerDied","Data":"f7bf44539d2bc6aee0660a4ab152c8bf56acce8b2a259723d0d8a7378b1acf4d"}
Oct 31 15:30:19.154349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.153943    4549 scope.go:122] "RemoveContainer" containerID="e1963ef41bd7ceb4dd15d710b930c6243d1e2bb85c81a3da5b0b70f17538994e"
Oct 31 15:30:19.155401 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.155367    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"
Oct 31 15:30:19.155401 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.155388    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg" event={"ID":"af54e62a-f0b8-4fc0-a302-f9a30388f012","Type":"ContainerDied","Data":"cd305a42ac4796fa9c0a9d1d4cd7fc8611acde32041cbeebbd51b83bcc625876"}
Oct 31 15:30:19.162161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.162142    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"
Oct 31 15:30:19.162271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.162144    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp" event={"ID":"27ae9101-f2e9-40c8-9043-a35beb67562b","Type":"ContainerDied","Data":"0d86d131feb47b1fdf2c1d70712ca11c18ac52640ce294f6a61d3ea616f5295b"}
Oct 31 15:30:19.167136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.167104    4549 scope.go:122] "RemoveContainer" containerID="ca66854213e7de043fc79ef8fa4856ae67160b1cacabb6efd53c3bdcf9a21a4b"
Oct 31 15:30:19.179389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.179372    4549 scope.go:122] "RemoveContainer" containerID="119854b0f110cf8baea10f0dba8870c725e46ea72caf92b69280bb398ef0f0d6"
Oct 31 15:30:19.325147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.325120    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:19.329785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.329763    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:19.329954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.329937    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891010-dszrn"]
Oct 31 15:30:19.422797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.422702    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" path="/var/lib/kubelet/pods/19b18dea-261b-4335-bf0e-8a9a7e2d9fdf/volumes"
Oct 31 15:30:19.442188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.442152    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:19.445485 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.445457    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-e72453-29891010-ckqkg"]
Oct 31 15:30:19.454525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.454490    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\" (UniqueName: \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\") pod \"fceb4c53-3254-4103-8fb8-d026078649d1\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") "
Oct 31 15:30:19.454744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.454728    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\") pod \"fceb4c53-3254-4103-8fb8-d026078649d1\" (UID: \"fceb4c53-3254-4103-8fb8-d026078649d1\") "
Oct 31 15:30:19.455091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.455068    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca" pod "fceb4c53-3254-4103-8fb8-d026078649d1" (UID: "fceb4c53-3254-4103-8fb8-d026078649d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:30:19.455343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.455175    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fceb4c53-3254-4103-8fb8-d026078649d1-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:19.456735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.456713    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m" pod "fceb4c53-3254-4103-8fb8-d026078649d1" (UID: "fceb4c53-3254-4103-8fb8-d026078649d1"). InnerVolumeSpecName "kube-api-access-fgz5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:30:19.559990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.559681    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:19.561988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.561959    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgz5m\" (UniqueName: \"kubernetes.io/projected/fceb4c53-3254-4103-8fb8-d026078649d1-kube-api-access-fgz5m\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:30:19.562357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:19.562330    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891010-49drp"]
Oct 31 15:30:20.181030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:20.181001    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"
Oct 31 15:30:20.181030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:20.181021    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf" event={"ID":"fceb4c53-3254-4103-8fb8-d026078649d1","Type":"ContainerDied","Data":"a9619418314514df6d6e1badab1b8a7b07dd88c504032e86f9b851030b6892f8"}
Oct 31 15:30:20.181630 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:20.181113    4549 scope.go:122] "RemoveContainer" containerID="9472d0dd5343700007d0014cecb06713b079e0df2b966c1f038c029e3740a139"
Oct 31 15:30:20.345393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:20.345356    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:20.348525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:20.348501    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-56bc96-29891010-cqnkf"]
Oct 31 15:30:21.418958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:21.418925    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" path="/var/lib/kubelet/pods/27ae9101-f2e9-40c8-9043-a35beb67562b/volumes"
Oct 31 15:30:21.419353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:21.419339    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" path="/var/lib/kubelet/pods/af54e62a-f0b8-4fc0-a302-f9a30388f012/volumes"
Oct 31 15:30:21.419749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:21.419734    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" path="/var/lib/kubelet/pods/fceb4c53-3254-4103-8fb8-d026078649d1/volumes"
Oct 31 15:30:37.014355 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:37.014317    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:30:44.373609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.373555    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:30:44.374101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374086    4549 cpu_manager.go:414] "containerMap: removing container" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374104    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374121    4549 cpu_manager.go:414] "containerMap: removing container" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374125    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374132    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374137    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374148    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374152    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374167    4549 cpu_manager.go:414] "containerMap: removing container" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374172    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374179    4549 cpu_manager.go:414] "containerMap: removing container" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374183    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374189    4549 cpu_manager.go:414] "containerMap: removing container" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374193    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374203    4549 cpu_manager.go:414] "containerMap: removing container" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374208    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374214    4549 cpu_manager.go:414] "containerMap: removing container" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374218    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374225    4549 cpu_manager.go:414] "containerMap: removing container" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374229    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374303    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374309    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374314    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374319    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374326    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374331    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374337    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374341    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374346    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:30:44.374380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.374357    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:30:44.377849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.377832    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.479182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.479146    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b25761-serving-cert\""
Oct 31 15:30:44.484252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.484218    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:30:44.484847 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.484829    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqss\" (UniqueName: \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.485009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.484985    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.485110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.485096    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.485246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.485230    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.586672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.586638    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.586868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.586690    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.586868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.586754    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.586868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.586857    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqss\" (UniqueName: \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.587038 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:44.586860    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-serving-cert: secret "maas-api-e2e-praxis-b-b25761-serving-cert" not found
Oct 31 15:30:44.587038 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:30:44.586949    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls podName:c7187818-5306-4728-bdca-721a379cc39b nodeName:}" failed. No retries permitted until 2026-10-31 15:30:45.086926897 +0000 UTC m=+2685.813207870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls") pod "maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" (UID: "c7187818-5306-4728-bdca-721a379cc39b") : secret "maas-api-e2e-praxis-b-b25761-serving-cert" not found
Oct 31 15:30:44.587521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.587492    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.589378 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.589355    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:44.649623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:44.649459    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqss\" (UniqueName: \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:45.093215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:45.093169    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:45.095839 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:45.095814    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:45.292377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:45.292336    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:45.670240 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:30:45.670200    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7187818_5306_4728_bdca_721a379cc39b.slice/crio-3617070a7c744726ae56cf26e1a88ced5b4301d95c4d517b49a6980b63546c44 WatchSource:0}: Error finding container 3617070a7c744726ae56cf26e1a88ced5b4301d95c4d517b49a6980b63546c44: Status 404 returned error can't find the container with id 3617070a7c744726ae56cf26e1a88ced5b4301d95c4d517b49a6980b63546c44
Oct 31 15:30:45.670653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:45.670376    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:30:46.392406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:46.392370    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" event={"ID":"c7187818-5306-4728-bdca-721a379cc39b","Type":"ContainerStarted","Data":"3617070a7c744726ae56cf26e1a88ced5b4301d95c4d517b49a6980b63546c44"}
Oct 31 15:30:47.402697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:47.402664    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" event={"ID":"c7187818-5306-4728-bdca-721a379cc39b","Type":"ContainerStarted","Data":"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"}
Oct 31 15:30:47.403211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:47.402777    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:30:47.522526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:47.522462    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" podStartSLOduration=2.653624528 podStartE2EDuration="3.522446995s" totalImagesPullingTime="868.822467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:30:47.519978512 +0000 UTC m=+2688.246259491" watchObservedRunningTime="2026-10-31 15:30:47.522446995 +0000 UTC m=+2688.248727966"
Oct 31 15:30:53.419527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:30:53.419500    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:31:02.879323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:02.879298    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:31:03.003888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:03.003864    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:31:07.013267 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:31:07.013226    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:31:18.496151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496118    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496661    4549 cpu_manager.go:414] "containerMap: removing container" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496673    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ae9101-f2e9-40c8-9043-a35beb67562b" containerName="cleanup"
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496689    4549 cpu_manager.go:414] "containerMap: removing container" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496695    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fceb4c53-3254-4103-8fb8-d026078649d1" containerName="cleanup"
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496778    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="af54e62a-f0b8-4fc0-a302-f9a30388f012" containerName="cleanup"
Oct 31 15:31:18.496801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.496791    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b18dea-261b-4335-bf0e-8a9a7e2d9fdf" containerName="cleanup"
Oct 31 15:31:18.501622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.501600    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.548220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.548192    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.548220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.548228    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnbm\" (UniqueName: \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.548420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.548250    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.548457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.548419    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.557523 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.557494    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:31:18.602063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.602033    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-2ad95a-serving-cert\""
Oct 31 15:31:18.650472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.650438    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.650703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.650616    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.650703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.650655    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnbm\" (UniqueName: \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.650703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.650682    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.650909 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:31:18.650889    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-serving-cert: secret "maas-api-e2e-autores-2ad95a-serving-cert" not found
Oct 31 15:31:18.650982 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:31:18.650973    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls podName:4871fbb6-366a-44ba-9031-7e89fae295a3 nodeName:}" failed. No retries permitted until 2026-10-31 15:31:19.150953004 +0000 UTC m=+2719.877233963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls") pod "maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" (UID: "4871fbb6-366a-44ba-9031-7e89fae295a3") : secret "maas-api-e2e-autores-2ad95a-serving-cert" not found
Oct 31 15:31:18.651734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.651707    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.653866 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.653843    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:18.709775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:18.709742    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnbm\" (UniqueName: \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:19.158547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:19.158507    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:19.161088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:19.161067    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") pod \"maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:19.417228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:19.417142    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:19.792535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:19.792502    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:31:19.794307 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:31:19.794279    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4871fbb6_366a_44ba_9031_7e89fae295a3.slice/crio-884fcfb246a2d8274a8fc2dfe4c3373c14bdcb45cbcaede24c22efa61d48d472 WatchSource:0}: Error finding container 884fcfb246a2d8274a8fc2dfe4c3373c14bdcb45cbcaede24c22efa61d48d472: Status 404 returned error can't find the container with id 884fcfb246a2d8274a8fc2dfe4c3373c14bdcb45cbcaede24c22efa61d48d472
Oct 31 15:31:20.643853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:20.643821    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" event={"ID":"4871fbb6-366a-44ba-9031-7e89fae295a3","Type":"ContainerStarted","Data":"884fcfb246a2d8274a8fc2dfe4c3373c14bdcb45cbcaede24c22efa61d48d472"}
Oct 31 15:31:21.654621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:21.654547    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" event={"ID":"4871fbb6-366a-44ba-9031-7e89fae295a3","Type":"ContainerStarted","Data":"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"}
Oct 31 15:31:21.655127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:21.654659    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:21.772177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:21.772123    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" podStartSLOduration=2.946201018 podStartE2EDuration="3.772106438s" totalImagesPullingTime="825.90542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:21.768892001 +0000 UTC m=+2722.495172976" watchObservedRunningTime="2026-10-31 15:31:21.772106438 +0000 UTC m=+2722.498387416"
Oct 31 15:31:27.666459 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:27.666431    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:31:37.012591 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:31:37.012529    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:31:37.285507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.285470    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:31:37.289631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.289599    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.341248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.341219    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2ad95a-openshift-default-dockercfg-6br7x\""
Oct 31 15:31:37.384373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384344    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384383    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384414    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384546 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384520    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384651    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lbn\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384690    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384785    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384817    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.384908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.384890    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486101    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lbn\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486176    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486294    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486338    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486397    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486425    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486464    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486927 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.486987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.486951    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.487108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.487087    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.487324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.487297    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.487393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.487321    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.487393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.487298    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.487775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.487741    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.490390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.490371    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.495602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.495192    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:31:37.495780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.495757    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.544233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.544154    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lbn\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.544527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.544510    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-7tknq\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:37.604061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:37.604027    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:38.047188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.047154    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:31:38.048399 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:31:38.048371    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82f2067_627a_46c6_9aeb_802b70087364.slice/crio-60e167d0082d7d8c073b82ba373bf9de5d6ceeee821cd6a752b52bd0b0bf8626 WatchSource:0}: Error finding container 60e167d0082d7d8c073b82ba373bf9de5d6ceeee821cd6a752b52bd0b0bf8626: Status 404 returned error can't find the container with id 60e167d0082d7d8c073b82ba373bf9de5d6ceeee821cd6a752b52bd0b0bf8626
Oct 31 15:31:38.050975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.050941    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:31:38.051051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.051009    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:31:38.051051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.051041    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:31:38.766423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.766384    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" event={"ID":"e82f2067-627a-46c6-9aeb-802b70087364","Type":"ContainerStarted","Data":"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"}
Oct 31 15:31:38.766663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.766437    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" event={"ID":"e82f2067-627a-46c6-9aeb-802b70087364","Type":"ContainerStarted","Data":"60e167d0082d7d8c073b82ba373bf9de5d6ceeee821cd6a752b52bd0b0bf8626"}
Oct 31 15:31:38.941305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:38.941237    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podStartSLOduration=1.941216941 podStartE2EDuration="1.941216941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:38.936156845 +0000 UTC m=+2739.662437852" watchObservedRunningTime="2026-10-31 15:31:38.941216941 +0000 UTC m=+2739.667497920"
Oct 31 15:31:39.608390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:39.608357    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:39.609086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:39.609055    4549 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 15:31:39.609315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:39.609288    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 15:31:40.606050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:40.605958    4549 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 15:31:40.606326 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:40.606296    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 15:31:41.606594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:41.605982    4549 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 15:31:41.606594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:41.606047    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 15:31:42.619996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:42.619956    4549 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 15:31:42.620757 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:42.620724    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 15:31:43.617989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:43.617946    4549 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 31 15:31:43.618301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:43.618269    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 31 15:31:44.610609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:44.610545    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:44.611105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:44.610991    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:31:44.611718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:31:44.611701    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:32:07.012921 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:32:07.012879    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:32:37.013379 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:32:37.013335    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:32:54.734131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:54.734093    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:32:54.734724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:54.734291    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" containerID="cri-o://5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65" gracePeriod=30
Oct 31 15:32:54.784227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:54.784191    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:32:54.784494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:54.784449    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" containerID="cri-o://81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510" gracePeriod=30
Oct 31 15:32:57.874602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:57.872836    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:32:57.874602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:57.873168    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" containerID="cri-o://10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773" gracePeriod=30
Oct 31 15:32:58.185651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.185519    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:32:58.185827 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.185757    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" podUID="5c2a3fa2-7156-4725-be48-5a4c1e157add" containerName="maas-api" containerID="cri-o://c32b1c4ae465c197490216d9303c410151b7d22bf8b876512db084b2595db21c" gracePeriod=30
Oct 31 15:32:58.357629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.357436    4549 generic.go:356] "Generic (PLEG): container finished" podID="5c2a3fa2-7156-4725-be48-5a4c1e157add" containerID="c32b1c4ae465c197490216d9303c410151b7d22bf8b876512db084b2595db21c" exitCode=0
Oct 31 15:32:58.357629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.357556    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" event={"ID":"5c2a3fa2-7156-4725-be48-5a4c1e157add","Type":"ContainerDied","Data":"c32b1c4ae465c197490216d9303c410151b7d22bf8b876512db084b2595db21c"}
Oct 31 15:32:58.572596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.570929    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:32:58.651729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.651690    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") pod \"5c2a3fa2-7156-4725-be48-5a4c1e157add\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") "
Oct 31 15:32:58.652070 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.652051    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\" (UniqueName: \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\") pod \"5c2a3fa2-7156-4725-be48-5a4c1e157add\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") "
Oct 31 15:32:58.652217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.652202    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\") pod \"5c2a3fa2-7156-4725-be48-5a4c1e157add\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") "
Oct 31 15:32:58.652341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.652325    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\") pod \"5c2a3fa2-7156-4725-be48-5a4c1e157add\" (UID: \"5c2a3fa2-7156-4725-be48-5a4c1e157add\") "
Oct 31 15:32:58.652586 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.652542    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca" pod "5c2a3fa2-7156-4725-be48-5a4c1e157add" (UID: "5c2a3fa2-7156-4725-be48-5a4c1e157add"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:32:58.653089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.653065    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2a3fa2-7156-4725-be48-5a4c1e157add-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:32:58.657752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.656408    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn" pod "5c2a3fa2-7156-4725-be48-5a4c1e157add" (UID: "5c2a3fa2-7156-4725-be48-5a4c1e157add"). InnerVolumeSpecName "kube-api-access-cq7fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:32:58.660694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.658176    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs" pod "5c2a3fa2-7156-4725-be48-5a4c1e157add" (UID: "5c2a3fa2-7156-4725-be48-5a4c1e157add"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:32:58.660694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.659030    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls" pod "5c2a3fa2-7156-4725-be48-5a4c1e157add" (UID: "5c2a3fa2-7156-4725-be48-5a4c1e157add"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:32:58.754328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.754287    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:32:58.754328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.754329    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq7fn\" (UniqueName: \"kubernetes.io/projected/5c2a3fa2-7156-4725-be48-5a4c1e157add-kube-api-access-cq7fn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:32:58.754629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:58.754345    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c2a3fa2-7156-4725-be48-5a4c1e157add-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:32:59.367978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:59.367945    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz" event={"ID":"5c2a3fa2-7156-4725-be48-5a4c1e157add","Type":"ContainerDied","Data":"f97b27fdfc71fbf56af03ab39daca93dac944bf2dd13c915eed29ce462297e38"}
Oct 31 15:32:59.368626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:59.367951    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"
Oct 31 15:32:59.368626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:59.368037    4549 scope.go:122] "RemoveContainer" containerID="c32b1c4ae465c197490216d9303c410151b7d22bf8b876512db084b2595db21c"
Oct 31 15:32:59.532062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:59.532023    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:32:59.535444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:32:59.535417    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5fccd7-5f95478765-svjvz"]
Oct 31 15:33:00.075984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.075953    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:33:00.141521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.141492    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.175781    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.175838    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.175902    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.175942    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.175983    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.176011    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.176241    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.176528    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.176605    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.176676 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.176651    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.178210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.177747    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.178210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178176    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:33:00.178470 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178446    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178762    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178797    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178833    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178867    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178915    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.179166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.178971    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.179463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179186    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.179463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179343    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.179581 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179509    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.179648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179603    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.179710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179660    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\") pod \"1c068373-ff33-411e-b3df-2f40f763af0e\" (UID: \"1c068373-ff33-411e-b3df-2f40f763af0e\") "
Oct 31 15:33:00.179765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179714    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.179821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.179764    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\") pod \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\" (UID: \"cc6c9515-1328-4f34-9c3d-3c9aa65e5250\") "
Oct 31 15:33:00.180854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180828    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180860    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180876    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180890    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180905    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180919    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180933    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.180955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.180947    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.181444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.181218    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.181832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.181796    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.181972 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.181947    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:33:00.182789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.182767    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "kube-api-access-8rh85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:00.182932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.182891    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:33:00.183130 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.183107    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:33:00.183240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.183175    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "kube-api-access-snzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:00.184722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.184696    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token" pod "cc6c9515-1328-4f34-9c3d-3c9aa65e5250" (UID: "cc6c9515-1328-4f34-9c3d-3c9aa65e5250"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:00.186057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.186036    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:00.186268 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.186246    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy" pod "1c068373-ff33-411e-b3df-2f40f763af0e" (UID: "1c068373-ff33-411e-b3df-2f40f763af0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282075    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282105    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c068373-ff33-411e-b3df-2f40f763af0e-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282114    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282123    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rh85\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-kube-api-access-8rh85\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282134    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282144    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c068373-ff33-411e-b3df-2f40f763af0e-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282152    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282161    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snzt2\" (UniqueName: \"kubernetes.io/projected/1c068373-ff33-411e-b3df-2f40f763af0e-kube-api-access-snzt2\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282170    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c068373-ff33-411e-b3df-2f40f763af0e-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.282536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.282178    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6c9515-1328-4f34-9c3d-3c9aa65e5250-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:00.377065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.377030    4549 generic.go:356] "Generic (PLEG): container finished" podID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerID="5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65" exitCode=0
Oct 31 15:33:00.377560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.377075    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"
Oct 31 15:33:00.377560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.377114    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" event={"ID":"cc6c9515-1328-4f34-9c3d-3c9aa65e5250","Type":"ContainerDied","Data":"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"}
Oct 31 15:33:00.377560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.377163    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" event={"ID":"cc6c9515-1328-4f34-9c3d-3c9aa65e5250","Type":"ContainerDied","Data":"3e473eb34ea121a7f736e616684a793c05c40e617ef035f56d25f6718c0b2916"}
Oct 31 15:33:00.377560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.377184    4549 scope.go:122] "RemoveContainer" containerID="5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"
Oct 31 15:33:00.378520 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.378499    4549 generic.go:356] "Generic (PLEG): container finished" podID="1c068373-ff33-411e-b3df-2f40f763af0e" containerID="81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510" exitCode=0
Oct 31 15:33:00.378616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.378541    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"
Oct 31 15:33:00.378675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.378610    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" event={"ID":"1c068373-ff33-411e-b3df-2f40f763af0e","Type":"ContainerDied","Data":"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"}
Oct 31 15:33:00.378675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.378662    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" event={"ID":"1c068373-ff33-411e-b3df-2f40f763af0e","Type":"ContainerDied","Data":"265de6c8ad2bb42f0c3e55a5a52de1b34a1bfa5a85d6dd4ae437769a109545f1"}
Oct 31 15:33:00.388808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.388794    4549 scope.go:122] "RemoveContainer" containerID="5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"
Oct 31 15:33:00.389241 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:00.389087    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65\": container with ID starting with 5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65 not found: ID does not exist" containerID="5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"
Oct 31 15:33:00.389241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.389116    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65"} err="failed to get container status \"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65\": rpc error: code = NotFound desc = could not find container \"5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65\": container with ID starting with 5ddf8fd57429f206b925c211f16c311ac45d278437fbc41552671b4a7a79ec65 not found: ID does not exist"
Oct 31 15:33:00.389241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.389139    4549 scope.go:122] "RemoveContainer" containerID="81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"
Oct 31 15:33:00.399124 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.399105    4549 scope.go:122] "RemoveContainer" containerID="81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"
Oct 31 15:33:00.399422 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:00.399405    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510\": container with ID starting with 81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510 not found: ID does not exist" containerID="81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"
Oct 31 15:33:00.399464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.399431    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510"} err="failed to get container status \"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510\": rpc error: code = NotFound desc = could not find container \"81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510\": container with ID starting with 81b7be474fb8d2e38622575c68b81349a6dc3a5cf577f92a058103b3211ff510 not found: ID does not exist"
Oct 31 15:33:00.544550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.544448    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:33:00.547654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.547628    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz"]
Oct 31 15:33:00.657095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.657062    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:33:00.660785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.660761    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv"]
Oct 31 15:33:00.988205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.988119    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:33:00.988205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:00.988176    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-hp2kz" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:33:01.037313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:01.037278    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:33:01.037491 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:01.037332    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-8s8pv" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:33:01.419982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:01.419948    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" path="/var/lib/kubelet/pods/1c068373-ff33-411e-b3df-2f40f763af0e/volumes"
Oct 31 15:33:01.420420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:01.420406    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2a3fa2-7156-4725-be48-5a4c1e157add" path="/var/lib/kubelet/pods/5c2a3fa2-7156-4725-be48-5a4c1e157add/volumes"
Oct 31 15:33:01.420922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:01.420907    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" path="/var/lib/kubelet/pods/cc6c9515-1328-4f34-9c3d-3c9aa65e5250/volumes"
Oct 31 15:33:02.963402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:02.963364    4549 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Oct 31 15:33:02.963780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:02.963417    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Oct 31 15:33:03.151499 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.151425    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 15:33:03.151712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.151687    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy" containerID="cri-o://edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef" gracePeriod=30
Oct 31 15:33:03.155066 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.155044    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:33:03.218426 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218389    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218462    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218500    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218535    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218653    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218714    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218771    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218827    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.218844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218836    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:03.219117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218861    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\") pod \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\" (UID: \"08ee2425-ccf4-42e6-a524-cff0f8a6e763\") "
Oct 31 15:33:03.219117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218881    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:03.219117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.218971    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:03.219117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219086    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:03.219406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219376    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.219406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219399    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.219545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219413    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.219545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219429    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.219545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.219496    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:33:03.221252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.221216    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "kube-api-access-4vpjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:03.221447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.221424    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:03.221723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.221703    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:03.221915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.221893    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo" pod "08ee2425-ccf4-42e6-a524-cff0f8a6e763" (UID: "08ee2425-ccf4-42e6-a524-cff0f8a6e763"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:33:03.321027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.320989    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.321027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.321029    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpjs\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-kube-api-access-4vpjs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.321235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.321045    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.321235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.321060    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.321235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.321073    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08ee2425-ccf4-42e6-a524-cff0f8a6e763-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:03.403960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.403866    4549 generic.go:356] "Generic (PLEG): container finished" podID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerID="10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773" exitCode=0
Oct 31 15:33:03.403960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.403913    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"
Oct 31 15:33:03.403960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.403955    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" event={"ID":"08ee2425-ccf4-42e6-a524-cff0f8a6e763","Type":"ContainerDied","Data":"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"}
Oct 31 15:33:03.404254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.404007    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg" event={"ID":"08ee2425-ccf4-42e6-a524-cff0f8a6e763","Type":"ContainerDied","Data":"c2abf2fb195ffe7f214b007550e638e77b5acdbb5b96ac2889d67474619b6979"}
Oct 31 15:33:03.404254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.404020    4549 scope.go:122] "RemoveContainer" containerID="10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"
Oct 31 15:33:03.416739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.416706    4549 scope.go:122] "RemoveContainer" containerID="10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"
Oct 31 15:33:03.417021 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:03.417003    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773\": container with ID starting with 10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773 not found: ID does not exist" containerID="10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"
Oct 31 15:33:03.417119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.417030    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773"} err="failed to get container status \"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773\": rpc error: code = NotFound desc = could not find container \"10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773\": container with ID starting with 10ad646415d171e90e0de207d918028e419506f5cc9e2255f0df04b46b3f3773 not found: ID does not exist"
Oct 31 15:33:03.569671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.569633    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:33:03.572263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:03.572242    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-425qg"]
Oct 31 15:33:05.420867 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:05.420825    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" path="/var/lib/kubelet/pods/08ee2425-ccf4-42e6-a524-cff0f8a6e763/volumes"
Oct 31 15:33:07.015642 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:07.015602    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:33:08.436708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.436375    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 15:33:08.442984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.442954    4549 generic.go:356] "Generic (PLEG): container finished" podID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerID="edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef" exitCode=0
Oct 31 15:33:08.443136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.443070    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"
Oct 31 15:33:08.443200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.443135    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerDied","Data":"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"}
Oct 31 15:33:08.443200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.443176    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx" event={"ID":"3dbb75fd-257d-4afa-9673-e8282b3da468","Type":"ContainerDied","Data":"39b2c528dc69539b71a0eef36ec69fbced46853051dc5a5479bacd707315d56e"}
Oct 31 15:33:08.443306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.443202    4549 scope.go:122] "RemoveContainer" containerID="edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"
Oct 31 15:33:08.453062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.453045    4549 scope.go:122] "RemoveContainer" containerID="6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"
Oct 31 15:33:08.462687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.462668    4549 scope.go:122] "RemoveContainer" containerID="edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"
Oct 31 15:33:08.463011 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:08.462991    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef\": container with ID starting with edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef not found: ID does not exist" containerID="edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"
Oct 31 15:33:08.463072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.463022    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef"} err="failed to get container status \"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef\": rpc error: code = NotFound desc = could not find container \"edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef\": container with ID starting with edea83daa5f08e7913b00b0aa2f85eab290161875882265394271d0c549d47ef not found: ID does not exist"
Oct 31 15:33:08.463072 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.463041    4549 scope.go:122] "RemoveContainer" containerID="6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"
Oct 31 15:33:08.463300 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:08.463281    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74\": container with ID starting with 6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74 not found: ID does not exist" containerID="6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"
Oct 31 15:33:08.463358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.463309    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74"} err="failed to get container status \"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74\": rpc error: code = NotFound desc = could not find container \"6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74\": container with ID starting with 6aaeab6b06bc3e40ffec4265afce6b35ca2d5a4ee42dc5636fed2b023df85a74 not found: ID does not exist"
Oct 31 15:33:08.591679 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591638    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.591856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591700    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.591856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591775    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.591856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591804    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.591856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591829    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.592074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.591968    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.592074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592027    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.592074 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592068    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.592305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592092    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:08.592305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592106    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:08.592305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592139    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\") pod \"3dbb75fd-257d-4afa-9673-e8282b3da468\" (UID: \"3dbb75fd-257d-4afa-9673-e8282b3da468\") "
Oct 31 15:33:08.592305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592255    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:08.592305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592277    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:08.592637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592611    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:33:08.592637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592630    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.592760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592648    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.592760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592691    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.592760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.592705    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.595107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.595080    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:33:08.595926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.595903    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "kube-api-access-bt6pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:08.595926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.595910    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:33:08.596714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.596689    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token" pod "3dbb75fd-257d-4afa-9673-e8282b3da468" (UID: "3dbb75fd-257d-4afa-9673-e8282b3da468"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:33:08.694254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.694179    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbb75fd-257d-4afa-9673-e8282b3da468-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.694254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.694216    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6pl\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-kube-api-access-bt6pl\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.694254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.694231    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.694254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.694245    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.694499 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.694260    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbb75fd-257d-4afa-9673-e8282b3da468-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:33:08.865093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.865047    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 15:33:08.868194 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:08.868168    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-bsnpx"]
Oct 31 15:33:09.421516 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.421473    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" path="/var/lib/kubelet/pods/3dbb75fd-257d-4afa-9673-e8282b3da468/volumes"
Oct 31 15:33:09.703959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.703871    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704352    4549 cpu_manager.go:414] "containerMap: removing container" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704363    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704375    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704382    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704399    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704404    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704410    4549 cpu_manager.go:414] "containerMap: removing container" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704415    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704425    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704429    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704441    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5c2a3fa2-7156-4725-be48-5a4c1e157add" containerName="maas-api"
Oct 31 15:33:09.704448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704447    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2a3fa2-7156-4725-be48-5a4c1e157add" containerName="maas-api"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704515    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2a3fa2-7156-4725-be48-5a4c1e157add" containerName="maas-api"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704522    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c068373-ff33-411e-b3df-2f40f763af0e" containerName="istio-proxy"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704531    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704537    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6c9515-1328-4f34-9c3d-3c9aa65e5250" containerName="istio-proxy"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704544    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbb75fd-257d-4afa-9673-e8282b3da468" containerName="istio-proxy"
Oct 31 15:33:09.705037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.704551    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ee2425-ccf4-42e6-a524-cff0f8a6e763" containerName="istio-proxy"
Oct 31 15:33:09.711729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.711700    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.798641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.776228    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:33:09.799934 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.799907    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"openshift-service-ca.crt\""
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810222    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810339    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810408    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810442    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znh9\" (UniqueName: \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810469    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.810708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.810494    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.813564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.813520    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"kube-root-ca.crt\""
Oct 31 15:33:09.840597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.840122    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"auth-test-model-aa864d-kserve-self-signed-certs\""
Oct 31 15:33:09.840597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.840263    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"default-dockercfg-6m78b\""
Oct 31 15:33:09.911631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911595    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.911800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911686    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.911800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911723    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znh9\" (UniqueName: \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.911800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911757    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.911952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911801    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.911952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.911927    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.912809 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.912719    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.913030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.912942    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.913100 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.913088    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.914871 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.914842    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.915623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.915608    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:09.970465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:09.970387    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znh9\" (UniqueName: \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\") pod \"auth-test-model-aa864d-kserve-694cf5f5fc-h88n8\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:10.034594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:10.034513    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:10.436120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:10.436089    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:33:10.437457 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:33:10.437419    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a484924_7247_4344_8904_a7d68943c851.slice/crio-004176aec98f0ed8ea964e6db97a4ffca2ab537b759166d0f6e5601b535e42ad WatchSource:0}: Error finding container 004176aec98f0ed8ea964e6db97a4ffca2ab537b759166d0f6e5601b535e42ad: Status 404 returned error can't find the container with id 004176aec98f0ed8ea964e6db97a4ffca2ab537b759166d0f6e5601b535e42ad
Oct 31 15:33:10.459625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:10.459563    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" event={"ID":"6a484924-7247-4344-8904-a7d68943c851","Type":"ContainerStarted","Data":"004176aec98f0ed8ea964e6db97a4ffca2ab537b759166d0f6e5601b535e42ad"}
Oct 31 15:33:11.467057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:11.467020    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" event={"ID":"6a484924-7247-4344-8904-a7d68943c851","Type":"ContainerStarted","Data":"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"}
Oct 31 15:33:11.581237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:11.581169    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" podStartSLOduration=2.581153161 podStartE2EDuration="2.581153161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:33:11.57989765 +0000 UTC m=+2832.306178628" watchObservedRunningTime="2026-10-31 15:33:11.581153161 +0000 UTC m=+2832.307434140"
Oct 31 15:33:12.035780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:12.035737    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:20.036144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:20.036100    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:20.049396 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:20.049356    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:20.541521 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:33:20.541491    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:33:37.015531 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:33:37.015488    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:34:07.011041 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:34:07.011000    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:34:11.284668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.284632    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:34:11.285294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.284891    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" podUID="c7187818-5306-4728-bdca-721a379cc39b" containerName="maas-api" containerID="cri-o://4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42" gracePeriod=30
Oct 31 15:34:11.601933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.601904    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:34:11.667645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.667605    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\") pod \"c7187818-5306-4728-bdca-721a379cc39b\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") "
Oct 31 15:34:11.667931 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.667915    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") pod \"c7187818-5306-4728-bdca-721a379cc39b\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") "
Oct 31 15:34:11.668513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.668491    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\") pod \"c7187818-5306-4728-bdca-721a379cc39b\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") "
Oct 31 15:34:11.669872 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.669757    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca" pod "c7187818-5306-4728-bdca-721a379cc39b" (UID: "c7187818-5306-4728-bdca-721a379cc39b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:34:11.670223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.670205    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\" (UniqueName: \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\") pod \"c7187818-5306-4728-bdca-721a379cc39b\" (UID: \"c7187818-5306-4728-bdca-721a379cc39b\") "
Oct 31 15:34:11.671192 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.671165    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7187818-5306-4728-bdca-721a379cc39b-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:34:11.679601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.678759    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss" pod "c7187818-5306-4728-bdca-721a379cc39b" (UID: "c7187818-5306-4728-bdca-721a379cc39b"). InnerVolumeSpecName "kube-api-access-hzqss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:34:11.697603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.697201    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs" pod "c7187818-5306-4728-bdca-721a379cc39b" (UID: "c7187818-5306-4728-bdca-721a379cc39b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:34:11.717591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.707398    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls" pod "c7187818-5306-4728-bdca-721a379cc39b" (UID: "c7187818-5306-4728-bdca-721a379cc39b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:34:11.772811 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.772771    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqss\" (UniqueName: \"kubernetes.io/projected/c7187818-5306-4728-bdca-721a379cc39b-kube-api-access-hzqss\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:34:11.772811 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.772808    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:34:11.773069 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.772822    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7187818-5306-4728-bdca-721a379cc39b-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:34:11.921421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.921328    4549 generic.go:356] "Generic (PLEG): container finished" podID="c7187818-5306-4728-bdca-721a379cc39b" containerID="4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42" exitCode=0
Oct 31 15:34:11.921421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.921382    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"
Oct 31 15:34:11.921421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.921393    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" event={"ID":"c7187818-5306-4728-bdca-721a379cc39b","Type":"ContainerDied","Data":"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"}
Oct 31 15:34:11.921667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.921446    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw" event={"ID":"c7187818-5306-4728-bdca-721a379cc39b","Type":"ContainerDied","Data":"3617070a7c744726ae56cf26e1a88ced5b4301d95c4d517b49a6980b63546c44"}
Oct 31 15:34:11.921667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.921475    4549 scope.go:122] "RemoveContainer" containerID="4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"
Oct 31 15:34:11.932729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.932642    4549 scope.go:122] "RemoveContainer" containerID="4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"
Oct 31 15:34:11.933033 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:34:11.933000    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42\": container with ID starting with 4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42 not found: ID does not exist" containerID="4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"
Oct 31 15:34:11.933151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:11.933038    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42"} err="failed to get container status \"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42\": rpc error: code = NotFound desc = could not find container \"4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42\": container with ID starting with 4793a1627573cb3e93c8626f75d4f2bd0d6f2e710a8f2ebc0eee1ae0c7877b42 not found: ID does not exist"
Oct 31 15:34:12.085936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:12.085900    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:34:12.090313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:12.090286    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b25761-7f98464c87-fnmcw"]
Oct 31 15:34:13.419870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:13.419836    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7187818-5306-4728-bdca-721a379cc39b" path="/var/lib/kubelet/pods/c7187818-5306-4728-bdca-721a379cc39b/volumes"
Oct 31 15:34:18.888704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.888671    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:34:18.889365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.889345    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c7187818-5306-4728-bdca-721a379cc39b" containerName="maas-api"
Oct 31 15:34:18.889365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.889365    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7187818-5306-4728-bdca-721a379cc39b" containerName="maas-api"
Oct 31 15:34:18.889503 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.889454    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7187818-5306-4728-bdca-721a379cc39b" containerName="maas-api"
Oct 31 15:34:18.894121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.894104    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:18.956603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.946744    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-24ca6e-openshift-default-dockercfg-848vm\""
Oct 31 15:34:18.968594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:18.968096    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:34:19.083110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.071301    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083541    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083628    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083728    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083762    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083789    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083851    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.083958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083910    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.084063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.083969    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwwc\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185340    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185390    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185471    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185502    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185524    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185590    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185643    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.185752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185693    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwwc\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.185756    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.186045    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186081 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.186064    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.186163    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.186215    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.186506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.186482    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.188276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.188250    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.188421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.188400    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.244362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.244299    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.245091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.244854    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwwc\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\") pod \"e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") " pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.511424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.511329    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:19.952374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.952344    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:34:19.953983 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:34:19.953948    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2afd94_2685_40bd_9270_cbffaa35926a.slice/crio-bd2551305fb88dc641dfbdfc479a058c6adc070f2810709ed2aacbe15123a4ec WatchSource:0}: Error finding container bd2551305fb88dc641dfbdfc479a058c6adc070f2810709ed2aacbe15123a4ec: Status 404 returned error can't find the container with id bd2551305fb88dc641dfbdfc479a058c6adc070f2810709ed2aacbe15123a4ec
Oct 31 15:34:19.957102 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.957058    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:34:19.957213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.957155    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:34:19.957213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.957197    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:34:19.991932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:19.991907    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerStarted","Data":"bd2551305fb88dc641dfbdfc479a058c6adc070f2810709ed2aacbe15123a4ec"}
Oct 31 15:34:21.000177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:21.000139    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerStarted","Data":"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"}
Oct 31 15:34:21.168046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:21.167992    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podStartSLOduration=3.167976679 podStartE2EDuration="3.167976679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:34:21.166644565 +0000 UTC m=+2901.892925546" watchObservedRunningTime="2026-10-31 15:34:21.167976679 +0000 UTC m=+2901.894257656"
Oct 31 15:34:21.512021 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:21.511987    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:21.519817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:21.519779    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:22.008101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:22.008072    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:22.009278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:34:22.009257    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:34:37.012766 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:34:37.012726    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:35:07.012502 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:35:07.012465    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:35:19.234608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.234557    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:35:19.235170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.234776    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerName="discovery" containerID="cri-o://0131da189f7977a47d4d17b804dd0a49f53a2aee98d7b9f4d485482f1b5d6035" gracePeriod=30
Oct 31 15:35:19.428559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.428528    4549 generic.go:356] "Generic (PLEG): container finished" podID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerID="0131da189f7977a47d4d17b804dd0a49f53a2aee98d7b9f4d485482f1b5d6035" exitCode=0
Oct 31 15:35:19.428773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.428697    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" event={"ID":"a1a77c56-4a4c-4b8f-8d20-82e667eb0509","Type":"ContainerDied","Data":"0131da189f7977a47d4d17b804dd0a49f53a2aee98d7b9f4d485482f1b5d6035"}
Oct 31 15:35:19.533541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.533513    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:35:19.723800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.723763    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.723985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.723817    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.723985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.723899    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.724088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.724021    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.724088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.724078    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.724512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.724479    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.724668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.724407    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:35:19.724668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.724554    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\") pod \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\" (UID: \"a1a77c56-4a4c-4b8f-8d20-82e667eb0509\") "
Oct 31 15:35:19.725355 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.725331    4549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-ca-configmap\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.727128 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727102    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "kube-api-access-cphtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:35:19.728604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727606    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:35:19.728604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727721    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:35:19.728604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727806    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:35:19.728604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727882    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:35:19.728604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.727906    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token" pod "a1a77c56-4a4c-4b8f-8d20-82e667eb0509" (UID: "a1a77c56-4a4c-4b8f-8d20-82e667eb0509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:35:19.826135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826101    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cphtm\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-kube-api-access-cphtm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.826135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826129    4549 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-local-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.826135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826140    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.826386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826149    4549 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-cacerts\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.826386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826157    4549 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-kubeconfig\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:19.826386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:19.826165    4549 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1a77c56-4a4c-4b8f-8d20-82e667eb0509-istio-csr-dns-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:20.437933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:20.437890    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"
Oct 31 15:35:20.437933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:20.437916    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5" event={"ID":"a1a77c56-4a4c-4b8f-8d20-82e667eb0509","Type":"ContainerDied","Data":"1fc19ba0f2ab3e8090e24ed291e55383b59d1b76fa076dbf3a8345ef3d4d1788"}
Oct 31 15:35:20.438525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:20.437979    4549 scope.go:122] "RemoveContainer" containerID="0131da189f7977a47d4d17b804dd0a49f53a2aee98d7b9f4d485482f1b5d6035"
Oct 31 15:35:20.653468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:20.653430    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:35:20.701487 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:20.701418    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-75qr5"]
Oct 31 15:35:21.419715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:21.419684    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" path="/var/lib/kubelet/pods/a1a77c56-4a4c-4b8f-8d20-82e667eb0509/volumes"
Oct 31 15:35:25.635241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.635207    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:35:25.635766 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.635749    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerName="discovery"
Oct 31 15:35:25.635819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.635767    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerName="discovery"
Oct 31 15:35:25.635886 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.635877    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a77c56-4a4c-4b8f-8d20-82e667eb0509" containerName="discovery"
Oct 31 15:35:25.640421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.640401    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.695654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.695621    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.695852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.695699    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zq8d\" (UniqueName: \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.695852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.695847    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.695997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.695936    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.743798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.743764    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-24ca6e-serving-cert\""
Oct 31 15:35:25.747067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.747044    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:35:25.797226 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.797195    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.797415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.797264    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zq8d\" (UniqueName: \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.797415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.797404    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.797590 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.797437    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.797696 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:35:25.797643    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-serving-cert: secret "maas-api-e2e-shared-a-w2-24ca6e-serving-cert" not found
Oct 31 15:35:25.797770 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:35:25.797727    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls podName:0549343d-f5ed-405d-bd93-4b40bf92da51 nodeName:}" failed. No retries permitted until 2026-10-31 15:35:26.297703885 +0000 UTC m=+2967.023984844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls") pod "maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" (UID: "0549343d-f5ed-405d-bd93-4b40bf92da51") : secret "maas-api-e2e-shared-a-w2-24ca6e-serving-cert" not found
Oct 31 15:35:25.798368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.798349    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.800419 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.800396    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:25.856263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:25.856231    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zq8d\" (UniqueName: \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:26.307506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:26.307469    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:26.310089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:26.310058    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:26.554119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:26.554083    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:26.935854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:26.935828    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:35:26.937067 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:35:26.937034    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0549343d_f5ed_405d_bd93_4b40bf92da51.slice/crio-2493184e24ebab1560625e8490228adfb612f2fe5d1f730d5c0b5b9f3f2ee9d7 WatchSource:0}: Error finding container 2493184e24ebab1560625e8490228adfb612f2fe5d1f730d5c0b5b9f3f2ee9d7: Status 404 returned error can't find the container with id 2493184e24ebab1560625e8490228adfb612f2fe5d1f730d5c0b5b9f3f2ee9d7
Oct 31 15:35:26.938890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:26.938870    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:35:27.486051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:27.486012    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" event={"ID":"0549343d-f5ed-405d-bd93-4b40bf92da51","Type":"ContainerStarted","Data":"2493184e24ebab1560625e8490228adfb612f2fe5d1f730d5c0b5b9f3f2ee9d7"}
Oct 31 15:35:28.494059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:28.494024    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" event={"ID":"0549343d-f5ed-405d-bd93-4b40bf92da51","Type":"ContainerStarted","Data":"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"}
Oct 31 15:35:28.494560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:28.494275    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:28.612453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:28.612399    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" podStartSLOduration=2.639039802 podStartE2EDuration="3.612385058s" totalImagesPullingTime="973.345256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:35:28.60948188 +0000 UTC m=+2969.335762860" watchObservedRunningTime="2026-10-31 15:35:28.612385058 +0000 UTC m=+2969.338666057"
Oct 31 15:35:34.507451 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:34.507421    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:35:37.016528 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:35:37.016488    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:35:49.231595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.231084    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:35:49.231595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.231342    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" podUID="4871fbb6-366a-44ba-9031-7e89fae295a3" containerName="maas-api" containerID="cri-o://ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570" gracePeriod=30
Oct 31 15:35:49.670823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.669850    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:35:49.676814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.675359    4549 generic.go:356] "Generic (PLEG): container finished" podID="4871fbb6-366a-44ba-9031-7e89fae295a3" containerID="ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570" exitCode=0
Oct 31 15:35:49.676814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.675486    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" event={"ID":"4871fbb6-366a-44ba-9031-7e89fae295a3","Type":"ContainerDied","Data":"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"}
Oct 31 15:35:49.676814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.675586    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8" event={"ID":"4871fbb6-366a-44ba-9031-7e89fae295a3","Type":"ContainerDied","Data":"884fcfb246a2d8274a8fc2dfe4c3373c14bdcb45cbcaede24c22efa61d48d472"}
Oct 31 15:35:49.676814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.675632    4549 scope.go:122] "RemoveContainer" containerID="ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"
Oct 31 15:35:49.676814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.675843    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"
Oct 31 15:35:49.692649 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.692603    4549 scope.go:122] "RemoveContainer" containerID="ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"
Oct 31 15:35:49.693114 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:35:49.693077    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570\": container with ID starting with ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570 not found: ID does not exist" containerID="ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"
Oct 31 15:35:49.693202 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.693139    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570"} err="failed to get container status \"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570\": rpc error: code = NotFound desc = could not find container \"ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570\": container with ID starting with ef944872799612fb192d3e0994e0d98a8e535f71387a0527f351a1b7884a1570 not found: ID does not exist"
Oct 31 15:35:49.737596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.736831    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\") pod \"4871fbb6-366a-44ba-9031-7e89fae295a3\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") "
Oct 31 15:35:49.737596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.736904    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") pod \"4871fbb6-366a-44ba-9031-7e89fae295a3\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") "
Oct 31 15:35:49.737596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.736973    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\" (UniqueName: \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\") pod \"4871fbb6-366a-44ba-9031-7e89fae295a3\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") "
Oct 31 15:35:49.737596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.737152    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\") pod \"4871fbb6-366a-44ba-9031-7e89fae295a3\" (UID: \"4871fbb6-366a-44ba-9031-7e89fae295a3\") "
Oct 31 15:35:49.737596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.737318    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca" pod "4871fbb6-366a-44ba-9031-7e89fae295a3" (UID: "4871fbb6-366a-44ba-9031-7e89fae295a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:35:49.738169 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.737630    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4871fbb6-366a-44ba-9031-7e89fae295a3-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:49.743220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.742627    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm" pod "4871fbb6-366a-44ba-9031-7e89fae295a3" (UID: "4871fbb6-366a-44ba-9031-7e89fae295a3"). InnerVolumeSpecName "kube-api-access-mpnbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:35:49.743220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.742737    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls" pod "4871fbb6-366a-44ba-9031-7e89fae295a3" (UID: "4871fbb6-366a-44ba-9031-7e89fae295a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:35:49.743220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.743147    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs" pod "4871fbb6-366a-44ba-9031-7e89fae295a3" (UID: "4871fbb6-366a-44ba-9031-7e89fae295a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:35:49.840646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.840423    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:49.840646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.840461    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4871fbb6-366a-44ba-9031-7e89fae295a3-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:49.840646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:49.840479    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpnbm\" (UniqueName: \"kubernetes.io/projected/4871fbb6-366a-44ba-9031-7e89fae295a3-kube-api-access-mpnbm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:35:50.118718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:50.118621    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:35:50.129986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:50.129939    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-2ad95a-c449d4fc9-q6tv8"]
Oct 31 15:35:51.421332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:51.421299    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4871fbb6-366a-44ba-9031-7e89fae295a3" path="/var/lib/kubelet/pods/4871fbb6-366a-44ba-9031-7e89fae295a3/volumes"
Oct 31 15:35:53.048724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.048690    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:35:53.049134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.048757    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:35:53.707838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.707797    4549 generic.go:356] "Generic (PLEG): container finished" podID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerID="8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec" exitCode=0
Oct 31 15:35:53.708053 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.707873    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerDied","Data":"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"}
Oct 31 15:35:53.708406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.708391    4549 scope.go:122] "RemoveContainer" containerID="8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"
Oct 31 15:35:53.710111 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.710059    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:35:53.710241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.710156    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:35:53.710241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:53.710188    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:35:54.738248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:54.738212    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerStarted","Data":"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"}
Oct 31 15:35:55.513598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:55.512013    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:35:55.513598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:55.512140    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 15:35:55.513598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:55.512198    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 15:35:56.512098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:56.512062    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 15:35:56.512795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:56.512765    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 15:35:57.520594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:57.516930    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 15:35:57.520594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:57.516993    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 15:35:58.519594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:58.518697    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 15:35:58.519594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:58.518779    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 15:35:59.523596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:59.512922    4549 patch_prober.go:29] interesting pod/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body=
Oct 31 15:35:59.523596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:59.513010    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused"
Oct 31 15:35:59.856443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:59.856331    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:35:59.856965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:35:59.856933    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy" containerID="cri-o://a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7" gracePeriod=30
Oct 31 15:36:00.515777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:00.515750    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:36:00.516136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:00.516115    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:36:00.516916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:00.516897    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:36:01.303039 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.303001    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:36:01.303517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.303503    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4871fbb6-366a-44ba-9031-7e89fae295a3" containerName="maas-api"
Oct 31 15:36:01.303589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.303518    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4871fbb6-366a-44ba-9031-7e89fae295a3" containerName="maas-api"
Oct 31 15:36:01.303639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.303630    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4871fbb6-366a-44ba-9031-7e89fae295a3" containerName="maas-api"
Oct 31 15:36:01.307094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.307072    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.359703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.359677    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-2cb1b7-openshift-default-dockercfg-m755m\""
Oct 31 15:36:01.375801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.375563    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:36:01.399106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399072    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399125    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399245    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399283    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399340    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjtcn\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399415    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399444    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399476    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.399704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.399505    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500340    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjtcn\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500435    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500464    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500508    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500537    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500728    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500764    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500811    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.500842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.500836    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.501217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.501181    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.501453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.501433    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.501809 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.501785    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.501955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.501792    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.502093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.502071    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.504199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.504154    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.504319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.504296    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.557882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.557800    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.558195 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.558171    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjtcn\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\") pod \"e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") " pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:01.621058 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:01.621022    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:02.058442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.058411    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:36:02.059966 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:36:02.059931    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3371c5_2f81_4bfc_bdd1_219bd5ab97b0.slice/crio-b68f91d5a34c48ea7f811a43e6bff2b2eb6cd13854984a73d85b684d6fc31bd0 WatchSource:0}: Error finding container b68f91d5a34c48ea7f811a43e6bff2b2eb6cd13854984a73d85b684d6fc31bd0: Status 404 returned error can't find the container with id b68f91d5a34c48ea7f811a43e6bff2b2eb6cd13854984a73d85b684d6fc31bd0
Oct 31 15:36:02.062695 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.062660    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:36:02.062806 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.062741    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:36:02.062806 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.062784    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:36:02.804502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.804455    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerStarted","Data":"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"}
Oct 31 15:36:02.804967 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.804518    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerStarted","Data":"b68f91d5a34c48ea7f811a43e6bff2b2eb6cd13854984a73d85b684d6fc31bd0"}
Oct 31 15:36:02.921672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.921610    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podStartSLOduration=1.921591477 podStartE2EDuration="1.921591477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:02.919836299 +0000 UTC m=+3003.646117288" watchObservedRunningTime="2026-10-31 15:36:02.921591477 +0000 UTC m=+3003.647872453"
Oct 31 15:36:02.925874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:02.925851    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:36:03.069985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:03.069912    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:36:03.621624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:03.621554    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:03.627342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:03.627316    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:03.811028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:03.811003    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:03.812075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:03.812055    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:36:05.138725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.138658    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:36:05.244028 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.243997    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244050    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244072    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244109    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244131    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244159    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244192    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244227    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244272    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\") pod \"e82f2067-627a-46c6-9aeb-802b70087364\" (UID: \"e82f2067-627a-46c6-9aeb-802b70087364\") "
Oct 31 15:36:05.244670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244356    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:36:05.244778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244736    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:36:05.244854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.244838    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.245078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.245052    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:36:05.245357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.245109    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:36:05.245357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.245248    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:36:05.246846 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.246823    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:36:05.247111 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.247085    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:36:05.247208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.247129    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:36:05.247208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.247154    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn" pod "e82f2067-627a-46c6-9aeb-802b70087364" (UID: "e82f2067-627a-46c6-9aeb-802b70087364"). InnerVolumeSpecName "kube-api-access-d4lbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:36:05.346427 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346384    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346427 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346425    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346439    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346455    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346470    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e82f2067-627a-46c6-9aeb-802b70087364-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346484    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e82f2067-627a-46c6-9aeb-802b70087364-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346497    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e82f2067-627a-46c6-9aeb-802b70087364-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.346671 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.346511    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4lbn\" (UniqueName: \"kubernetes.io/projected/e82f2067-627a-46c6-9aeb-802b70087364-kube-api-access-d4lbn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:36:05.825148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.825116    4549 generic.go:356] "Generic (PLEG): container finished" podID="e82f2067-627a-46c6-9aeb-802b70087364" containerID="a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7" exitCode=0
Oct 31 15:36:05.825323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.825183    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"
Oct 31 15:36:05.825323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.825203    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" event={"ID":"e82f2067-627a-46c6-9aeb-802b70087364","Type":"ContainerDied","Data":"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"}
Oct 31 15:36:05.825323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.825279    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq" event={"ID":"e82f2067-627a-46c6-9aeb-802b70087364","Type":"ContainerDied","Data":"60e167d0082d7d8c073b82ba373bf9de5d6ceeee821cd6a752b52bd0b0bf8626"}
Oct 31 15:36:05.825323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.825298    4549 scope.go:122] "RemoveContainer" containerID="a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"
Oct 31 15:36:05.835908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.835884    4549 scope.go:122] "RemoveContainer" containerID="a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"
Oct 31 15:36:05.836202 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:36:05.836178    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7\": container with ID starting with a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7 not found: ID does not exist" containerID="a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"
Oct 31 15:36:05.836279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:05.836211    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7"} err="failed to get container status \"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7\": rpc error: code = NotFound desc = could not find container \"a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7\": container with ID starting with a5ece838647a662c3ae3580d3cb153ce6ccc10d8d38bd5b485121900476691e7 not found: ID does not exist"
Oct 31 15:36:06.040814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:06.040776    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:36:06.089962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:06.089876    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-7tknq"]
Oct 31 15:36:07.013592 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:36:07.013543    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:36:07.418781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:36:07.418752    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82f2067-627a-46c6-9aeb-802b70087364" path="/var/lib/kubelet/pods/e82f2067-627a-46c6-9aeb-802b70087364/volumes"
Oct 31 15:36:37.016865 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:36:37.016784    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:37:07.013413 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:07.013376    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:37:08.374516 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.374483    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:37:08.375054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.374972    4549 cpu_manager.go:414] "containerMap: removing container" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy"
Oct 31 15:37:08.375054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.374983    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy"
Oct 31 15:37:08.375131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.375084    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82f2067-627a-46c6-9aeb-802b70087364" containerName="istio-proxy"
Oct 31 15:37:08.379512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.379495    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.436698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.436663    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:37:08.480080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.480053    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-2cb1b7-serving-cert\""
Oct 31 15:37:08.529404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.529036    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.529404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.529095    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.529404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.529204    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr7dz\" (UniqueName: \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.529404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.529264    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.633361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.633268    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.633361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.633327    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.633601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.633438    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr7dz\" (UniqueName: \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.633601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.633505    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.633740 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:08.633682    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-serving-cert: secret "maas-api-e2e-explicit-2cb1b7-serving-cert" not found
Oct 31 15:37:08.633804 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:08.633742    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls podName:f4b65214-e014-4629-8b23-a511285285a2 nodeName:}" failed. No retries permitted until 2026-10-31 15:37:09.133722857 +0000 UTC m=+3069.860003813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls") pod "maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" (UID: "f4b65214-e014-4629-8b23-a511285285a2") : secret "maas-api-e2e-explicit-2cb1b7-serving-cert" not found
Oct 31 15:37:08.634150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.634123    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.636496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.636453    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:08.694746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:08.694721    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr7dz\" (UniqueName: \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:09.140852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:09.140798    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:09.143515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:09.143491    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") pod \"maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:09.293825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:09.293783    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:09.671796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:09.671766    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:37:09.672782 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:09.672755    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b65214_e014_4629_8b23_a511285285a2.slice/crio-5c76570c1d4cbd3d3ffe8cae1f44c1cedf68173c2886d952db81ad40195c7ebf WatchSource:0}: Error finding container 5c76570c1d4cbd3d3ffe8cae1f44c1cedf68173c2886d952db81ad40195c7ebf: Status 404 returned error can't find the container with id 5c76570c1d4cbd3d3ffe8cae1f44c1cedf68173c2886d952db81ad40195c7ebf
Oct 31 15:37:10.278101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:10.278063    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" event={"ID":"f4b65214-e014-4629-8b23-a511285285a2","Type":"ContainerStarted","Data":"5c76570c1d4cbd3d3ffe8cae1f44c1cedf68173c2886d952db81ad40195c7ebf"}
Oct 31 15:37:11.285530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:11.285489    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" event={"ID":"f4b65214-e014-4629-8b23-a511285285a2","Type":"ContainerStarted","Data":"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"}
Oct 31 15:37:11.285979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:11.285718    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:11.399313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:11.399253    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" podStartSLOduration=2.5349153380000002 podStartE2EDuration="3.399235915s" totalImagesPullingTime="864.320577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:11.39868047 +0000 UTC m=+3072.124961448" watchObservedRunningTime="2026-10-31 15:37:11.399235915 +0000 UTC m=+3072.125516893"
Oct 31 15:37:17.296682 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:17.296651    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:37:22.073819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.073782    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-48532e"]
Oct 31 15:37:22.077742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.077722    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:22.091502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.091481    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt476\" (UniqueName: \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\") pod \"test-curl-7588-48532e\" (UID: \"f6fdc3da-5bf8-4576-9721-025721c3ea7b\") " pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:22.131944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.131915    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-48532e"]
Oct 31 15:37:22.177661 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.177621    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\""
Oct 31 15:37:22.192262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.192231    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt476\" (UniqueName: \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\") pod \"test-curl-7588-48532e\" (UID: \"f6fdc3da-5bf8-4576-9721-025721c3ea7b\") " pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:22.250178 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.250143    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt476\" (UniqueName: \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\") pod \"test-curl-7588-48532e\" (UID: \"f6fdc3da-5bf8-4576-9721-025721c3ea7b\") " pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:22.389752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.389674    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:22.762013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:22.761988    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-48532e"]
Oct 31 15:37:22.762963 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:22.762937    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6fdc3da_5bf8_4576_9721_025721c3ea7b.slice/crio-a648a9da3d4cc6e74345363a3502a8ba7370b763e6aab74e6b29b29a90bb6a16 WatchSource:0}: Error finding container a648a9da3d4cc6e74345363a3502a8ba7370b763e6aab74e6b29b29a90bb6a16: Status 404 returned error can't find the container with id a648a9da3d4cc6e74345363a3502a8ba7370b763e6aab74e6b29b29a90bb6a16
Oct 31 15:37:23.364215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:23.364172    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-48532e" event={"ID":"f6fdc3da-5bf8-4576-9721-025721c3ea7b","Type":"ContainerStarted","Data":"fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def"}
Oct 31 15:37:23.364654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:23.364227    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-48532e" event={"ID":"f6fdc3da-5bf8-4576-9721-025721c3ea7b","Type":"ContainerStarted","Data":"a648a9da3d4cc6e74345363a3502a8ba7370b763e6aab74e6b29b29a90bb6a16"}
Oct 31 15:37:23.486274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:23.486218    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7588-48532e" podStartSLOduration=1.4862044509999999 podStartE2EDuration="1.486204451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:23.484863275 +0000 UTC m=+3084.211144256" watchObservedRunningTime="2026-10-31 15:37:23.486204451 +0000 UTC m=+3084.212485429"
Oct 31 15:37:25.019158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.019127    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-48532e"]
Oct 31 15:37:25.023256 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.023223    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-48532e"]
Oct 31 15:37:25.377440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.377347    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7588-48532e" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" containerName="test-curl-7588-48532e" containerID="cri-o://fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def" gracePeriod=2
Oct 31 15:37:25.428000 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:25.427967    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-48532e\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:25.448073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.448042    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-59cd93"]
Oct 31 15:37:25.448526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.448513    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" containerName="test-curl-7588-48532e"
Oct 31 15:37:25.448593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.448528    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" containerName="test-curl-7588-48532e"
Oct 31 15:37:25.448672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.448661    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" containerName="test-curl-7588-48532e"
Oct 31 15:37:25.451972 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.451953    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:25.509202 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.509172    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-59cd93"]
Oct 31 15:37:25.531423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.531355    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9cf\" (UniqueName: \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\") pod \"test-curl-7588-59cd93\" (UID: \"f3f3c94b-783b-4f15-a3b8-875c28d01b36\") " pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:25.632746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.632644    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9cf\" (UniqueName: \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\") pod \"test-curl-7588-59cd93\" (UID: \"f3f3c94b-783b-4f15-a3b8-875c28d01b36\") " pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:25.688916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.688888    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9cf\" (UniqueName: \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\") pod \"test-curl-7588-59cd93\" (UID: \"f3f3c94b-783b-4f15-a3b8-875c28d01b36\") " pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:25.764225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:25.764190    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:26.142121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:26.142089    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-59cd93"]
Oct 31 15:37:26.385642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:26.385608    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-59cd93" event={"ID":"f3f3c94b-783b-4f15-a3b8-875c28d01b36","Type":"ContainerStarted","Data":"c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187"}
Oct 31 15:37:26.385834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:26.385656    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-59cd93" event={"ID":"f3f3c94b-783b-4f15-a3b8-875c28d01b36","Type":"ContainerStarted","Data":"9268c8e0ef628d6bc75645ff42495d67d8f128b07fd4ed7a7cd09b68e2db2cb8"}
Oct 31 15:37:26.496907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:26.496797    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7588-59cd93" podStartSLOduration=1.4967835059999999 podStartE2EDuration="1.496783506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:26.495821003 +0000 UTC m=+3087.222101984" watchObservedRunningTime="2026-10-31 15:37:26.496783506 +0000 UTC m=+3087.223064478"
Oct 31 15:37:27.591511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.591485    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:27.654777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.654689    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\" (UniqueName: \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\") pod \"f6fdc3da-5bf8-4576-9721-025721c3ea7b\" (UID: \"f6fdc3da-5bf8-4576-9721-025721c3ea7b\") "
Oct 31 15:37:27.656911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.656885    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476" pod "f6fdc3da-5bf8-4576-9721-025721c3ea7b" (UID: "f6fdc3da-5bf8-4576-9721-025721c3ea7b"). InnerVolumeSpecName "kube-api-access-bt476". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:27.755958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.755916    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt476\" (UniqueName: \"kubernetes.io/projected/f6fdc3da-5bf8-4576-9721-025721c3ea7b-kube-api-access-bt476\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:27.851438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.851404    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-59cd93"]
Oct 31 15:37:27.855725 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:27.855695    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-59cd93"]
Oct 31 15:37:28.003340 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.003252    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-d53c8b"]
Oct 31 15:37:28.003786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.003771    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" containerName="test-curl-7588-59cd93"
Oct 31 15:37:28.003854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.003788    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" containerName="test-curl-7588-59cd93"
Oct 31 15:37:28.003893 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.003880    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" containerName="test-curl-7588-59cd93"
Oct 31 15:37:28.007163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.007146    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:28.059411 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.059364    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htzw\" (UniqueName: \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\") pod \"test-curl-7588-d53c8b\" (UID: \"e83118f1-d12c-4235-ae51-508b3915e88e\") " pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:28.062899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.062875    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-d53c8b"]
Oct 31 15:37:28.160666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.160633    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htzw\" (UniqueName: \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\") pod \"test-curl-7588-d53c8b\" (UID: \"e83118f1-d12c-4235-ae51-508b3915e88e\") " pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:28.216976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.216945    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htzw\" (UniqueName: \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\") pod \"test-curl-7588-d53c8b\" (UID: \"e83118f1-d12c-4235-ae51-508b3915e88e\") " pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:28.319268 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.319232    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:28.401298 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.401266    4549 generic.go:356] "Generic (PLEG): container finished" podID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" containerID="fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def" exitCode=137
Oct 31 15:37:28.401551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.401534    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7588-59cd93" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" containerName="test-curl-7588-59cd93" containerID="cri-o://c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187" gracePeriod=2
Oct 31 15:37:28.401875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.401858    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:28.404482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.403702    4549 scope.go:122] "RemoveContainer" containerID="fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def"
Oct 31 15:37:28.421184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.421159    4549 scope.go:122] "RemoveContainer" containerID="fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def"
Oct 31 15:37:28.421503 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:28.421483    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def\": container with ID starting with fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def not found: ID does not exist" containerID="fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def"
Oct 31 15:37:28.421596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.421514    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def"} err="failed to get container status \"fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def\": rpc error: code = NotFound desc = could not find container \"fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def\": container with ID starting with fa1d8e1f7de08d744f93eb84ddcd0d396c06376ccc44c8f5dbaf1fd648b35def not found: ID does not exist"
Oct 31 15:37:28.455363 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:28.455325    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:28.506061 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:28.506018    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:28.556507 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:28.556471    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-48532e\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:28.695422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:28.695393    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-d53c8b"]
Oct 31 15:37:28.696604 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:28.696553    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode83118f1_d12c_4235_ae51_508b3915e88e.slice/crio-e6c45a1e6079cdc4260e5c958b912bd27a29300492d0c0419225cfc7d287aecc WatchSource:0}: Error finding container e6c45a1e6079cdc4260e5c958b912bd27a29300492d0c0419225cfc7d287aecc: Status 404 returned error can't find the container with id e6c45a1e6079cdc4260e5c958b912bd27a29300492d0c0419225cfc7d287aecc
Oct 31 15:37:29.409200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:29.409159    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-d53c8b" event={"ID":"e83118f1-d12c-4235-ae51-508b3915e88e","Type":"ContainerStarted","Data":"85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e"}
Oct 31 15:37:29.409423 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:29.409215    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-d53c8b" event={"ID":"e83118f1-d12c-4235-ae51-508b3915e88e","Type":"ContainerStarted","Data":"e6c45a1e6079cdc4260e5c958b912bd27a29300492d0c0419225cfc7d287aecc"}
Oct 31 15:37:29.419682 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:29.419648    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" path="/var/lib/kubelet/pods/f6fdc3da-5bf8-4576-9721-025721c3ea7b/volumes"
Oct 31 15:37:29.460027 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:29.459995    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-48532e\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f6fdc3da-5bf8-4576-9721-025721c3ea7b" pod="opendatahub/test-curl-7588-48532e"
Oct 31 15:37:29.572332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:29.572275    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7588-d53c8b" podStartSLOduration=2.572261932 podStartE2EDuration="2.572261932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:29.569273957 +0000 UTC m=+3090.295554933" watchObservedRunningTime="2026-10-31 15:37:29.572261932 +0000 UTC m=+3090.298542901"
Oct 31 15:37:29.619631 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:29.619594    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:30.641052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:30.641027    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:30.691837 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:30.691727    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:30.691980 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:30.691951    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\" (UniqueName: \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\") pod \"f3f3c94b-783b-4f15-a3b8-875c28d01b36\" (UID: \"f3f3c94b-783b-4f15-a3b8-875c28d01b36\") "
Oct 31 15:37:30.694285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:30.694256    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf" pod "f3f3c94b-783b-4f15-a3b8-875c28d01b36" (UID: "f3f3c94b-783b-4f15-a3b8-875c28d01b36"). InnerVolumeSpecName "kube-api-access-7r9cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:30.793833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:30.793793    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9cf\" (UniqueName: \"kubernetes.io/projected/f3f3c94b-783b-4f15-a3b8-875c28d01b36-kube-api-access-7r9cf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:31.041135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.040721    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-d53c8b"]
Oct 31 15:37:31.044547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.044513    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-d53c8b"]
Oct 31 15:37:31.206203 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.206167    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-260562"]
Oct 31 15:37:31.206901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.206880    4549 cpu_manager.go:414] "containerMap: removing container" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" containerName="test-curl-7588-d53c8b"
Oct 31 15:37:31.207007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.206902    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" containerName="test-curl-7588-d53c8b"
Oct 31 15:37:31.207084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.207072    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" containerName="test-curl-7588-d53c8b"
Oct 31 15:37:31.210923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.210901    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:31.261542 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:31.261501    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:31.261838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.261819    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-260562"]
Oct 31 15:37:31.299368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.299265    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lb4t\" (UniqueName: \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\") pod \"test-curl-7588-260562\" (UID: \"44c8b34c-f196-44de-be8e-5da55efef0bf\") " pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:31.400947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.400905    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lb4t\" (UniqueName: \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\") pod \"test-curl-7588-260562\" (UID: \"44c8b34c-f196-44de-be8e-5da55efef0bf\") " pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:31.422801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.422760    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" path="/var/lib/kubelet/pods/f3f3c94b-783b-4f15-a3b8-875c28d01b36/volumes"
Oct 31 15:37:31.426249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.426219    4549 generic.go:356] "Generic (PLEG): container finished" podID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" containerID="c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187" exitCode=137
Oct 31 15:37:31.426405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.426263    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:31.426405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.426359    4549 scope.go:122] "RemoveContainer" containerID="c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187"
Oct 31 15:37:31.426687 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.426653    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7588-d53c8b" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" containerName="test-curl-7588-d53c8b" containerID="cri-o://85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e" gracePeriod=2
Oct 31 15:37:31.437777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.437682    4549 scope.go:122] "RemoveContainer" containerID="c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187"
Oct 31 15:37:31.438074 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:31.438047    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187\": container with ID starting with c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187 not found: ID does not exist" containerID="c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187"
Oct 31 15:37:31.438163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.438075    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187"} err="failed to get container status \"c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187\": rpc error: code = NotFound desc = could not find container \"c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187\": container with ID starting with c857d5da384dc49f96adc3758464bb072dbb004d16a698ca58bfa54a99f5c187 not found: ID does not exist"
Oct 31 15:37:31.457804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.457771    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lb4t\" (UniqueName: \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\") pod \"test-curl-7588-260562\" (UID: \"44c8b34c-f196-44de-be8e-5da55efef0bf\") " pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:31.477634 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:31.477595    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:31.525163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.525128    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:31.528430 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:31.528398    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:31.923062 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:31.923034    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-260562"]
Oct 31 15:37:31.924275 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:31.924246    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44c8b34c_f196_44de_be8e_5da55efef0bf.slice/crio-f888a233b21a9a506487da5d95687fe61f8337a8cfcbcae4539b51554fa0d9d1 WatchSource:0}: Error finding container f888a233b21a9a506487da5d95687fe61f8337a8cfcbcae4539b51554fa0d9d1: Status 404 returned error can't find the container with id f888a233b21a9a506487da5d95687fe61f8337a8cfcbcae4539b51554fa0d9d1
Oct 31 15:37:32.435125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:32.435085    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-260562" event={"ID":"44c8b34c-f196-44de-be8e-5da55efef0bf","Type":"ContainerStarted","Data":"56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c"}
Oct 31 15:37:32.435356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:32.435145    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-260562" event={"ID":"44c8b34c-f196-44de-be8e-5da55efef0bf","Type":"ContainerStarted","Data":"f888a233b21a9a506487da5d95687fe61f8337a8cfcbcae4539b51554fa0d9d1"}
Oct 31 15:37:32.548905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:32.548855    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7588-260562" podStartSLOduration=1.548842726 podStartE2EDuration="1.548842726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:32.545426567 +0000 UTC m=+3093.271707546" watchObservedRunningTime="2026-10-31 15:37:32.548842726 +0000 UTC m=+3093.275123704"
Oct 31 15:37:32.596174 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:32.596134    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-59cd93\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="f3f3c94b-783b-4f15-a3b8-875c28d01b36" pod="opendatahub/test-curl-7588-59cd93"
Oct 31 15:37:33.671615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.671559    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:33.727063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.726972    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\" (UniqueName: \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\") pod \"e83118f1-d12c-4235-ae51-508b3915e88e\" (UID: \"e83118f1-d12c-4235-ae51-508b3915e88e\") "
Oct 31 15:37:33.729306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.729280    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw" pod "e83118f1-d12c-4235-ae51-508b3915e88e" (UID: "e83118f1-d12c-4235-ae51-508b3915e88e"). InnerVolumeSpecName "kube-api-access-4htzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:33.828211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.828169    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htzw\" (UniqueName: \"kubernetes.io/projected/e83118f1-d12c-4235-ae51-508b3915e88e-kube-api-access-4htzw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:33.896939 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.896905    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-260562"]
Oct 31 15:37:33.900981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:33.900953    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-260562"]
Oct 31 15:37:34.045095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.045063    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7588-fe160e"]
Oct 31 15:37:34.045859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.045839    4549 cpu_manager.go:414] "containerMap: removing container" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" containerName="test-curl-7588-260562"
Oct 31 15:37:34.045926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.045863    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" containerName="test-curl-7588-260562"
Oct 31 15:37:34.046007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.045995    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" containerName="test-curl-7588-260562"
Oct 31 15:37:34.049441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.049426    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:34.101910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.101878    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-fe160e"]
Oct 31 15:37:34.131769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.131735    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xh69\" (UniqueName: \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\") pod \"test-curl-7588-fe160e\" (UID: \"d4708017-d175-486e-a29f-fcba166fa8ac\") " pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:34.232653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.232620    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xh69\" (UniqueName: \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\") pod \"test-curl-7588-fe160e\" (UID: \"d4708017-d175-486e-a29f-fcba166fa8ac\") " pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:34.289301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.289271    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xh69\" (UniqueName: \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\") pod \"test-curl-7588-fe160e\" (UID: \"d4708017-d175-486e-a29f-fcba166fa8ac\") " pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:34.361548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.361475    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:34.454314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.454277    4549 generic.go:356] "Generic (PLEG): container finished" podID="e83118f1-d12c-4235-ae51-508b3915e88e" containerID="85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e" exitCode=137
Oct 31 15:37:34.454475 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.454349    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:34.454475 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.454368    4549 scope.go:122] "RemoveContainer" containerID="85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e"
Oct 31 15:37:34.454681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.454662    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7588-260562" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" containerName="test-curl-7588-260562" containerID="cri-o://56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c" gracePeriod=2
Oct 31 15:37:34.467408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.467387    4549 scope.go:122] "RemoveContainer" containerID="85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e"
Oct 31 15:37:34.467735 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:34.467716    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e\": container with ID starting with 85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e not found: ID does not exist" containerID="85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e"
Oct 31 15:37:34.467811 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.467746    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e"} err="failed to get container status \"85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e\": rpc error: code = NotFound desc = could not find container \"85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e\": container with ID starting with 85e2272983ba94fc8515b6ba44b55230569fa4855fb3f8ace3dbf5d3d259ae7e not found: ID does not exist"
Oct 31 15:37:34.505887 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:34.505847    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-260562\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:34.562596 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:34.562234    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-d53c8b\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" pod="opendatahub/test-curl-7588-d53c8b"
Oct 31 15:37:34.612859 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:34.612768    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-260562\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:34.737591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:34.737546    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7588-fe160e"]
Oct 31 15:37:34.738979 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:34.738942    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4708017_d175_486e_a29f_fcba166fa8ac.slice/crio-7a82c40f089c1bff67de833afd8a57f14eefa675c883c4a8245515fb4cb9ae67 WatchSource:0}: Error finding container 7a82c40f089c1bff67de833afd8a57f14eefa675c883c4a8245515fb4cb9ae67: Status 404 returned error can't find the container with id 7a82c40f089c1bff67de833afd8a57f14eefa675c883c4a8245515fb4cb9ae67
Oct 31 15:37:35.425269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:35.425234    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83118f1-d12c-4235-ae51-508b3915e88e" path="/var/lib/kubelet/pods/e83118f1-d12c-4235-ae51-508b3915e88e/volumes"
Oct 31 15:37:35.461825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:35.461794    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-fe160e" event={"ID":"d4708017-d175-486e-a29f-fcba166fa8ac","Type":"ContainerStarted","Data":"2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3"}
Oct 31 15:37:35.461993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:35.461837    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7588-fe160e" event={"ID":"d4708017-d175-486e-a29f-fcba166fa8ac","Type":"ContainerStarted","Data":"7a82c40f089c1bff67de833afd8a57f14eefa675c883c4a8245515fb4cb9ae67"}
Oct 31 15:37:35.574086 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:35.574035    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7588-fe160e" podStartSLOduration=1.574022427 podStartE2EDuration="1.574022427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:35.572762151 +0000 UTC m=+3096.299043130" watchObservedRunningTime="2026-10-31 15:37:35.574022427 +0000 UTC m=+3096.300303405"
Oct 31 15:37:36.768969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:36.768945    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:36.803100 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:36.803047    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\" (UniqueName: \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\") pod \"44c8b34c-f196-44de-be8e-5da55efef0bf\" (UID: \"44c8b34c-f196-44de-be8e-5da55efef0bf\") "
Oct 31 15:37:36.828270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:36.827140    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t" pod "44c8b34c-f196-44de-be8e-5da55efef0bf" (UID: "44c8b34c-f196-44de-be8e-5da55efef0bf"). InnerVolumeSpecName "kube-api-access-2lb4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:36.912596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:36.912541    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lb4t\" (UniqueName: \"kubernetes.io/projected/44c8b34c-f196-44de-be8e-5da55efef0bf-kube-api-access-2lb4t\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:37.032970 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:37.032106    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:37:37.144813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.144696    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7588-fe160e"]
Oct 31 15:37:37.146135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.146107    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7588-fe160e"]
Oct 31 15:37:37.428739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.428657    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c8b34c-f196-44de-be8e-5da55efef0bf" path="/var/lib/kubelet/pods/44c8b34c-f196-44de-be8e-5da55efef0bf/volumes"
Oct 31 15:37:37.513597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.494972    4549 generic.go:356] "Generic (PLEG): container finished" podID="44c8b34c-f196-44de-be8e-5da55efef0bf" containerID="56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c" exitCode=137
Oct 31 15:37:37.513597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.495025    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-260562"
Oct 31 15:37:37.513597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.495057    4549 scope.go:122] "RemoveContainer" containerID="56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c"
Oct 31 15:37:37.513597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.495426    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7588-fe160e" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" containerName="test-curl-7588-fe160e" containerID="cri-o://2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3" gracePeriod=2
Oct 31 15:37:37.535448 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.535417    4549 scope.go:122] "RemoveContainer" containerID="56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c"
Oct 31 15:37:37.537122 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:37.537089    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c\": container with ID starting with 56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c not found: ID does not exist" containerID="56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c"
Oct 31 15:37:37.537255 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:37.537134    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c"} err="failed to get container status \"56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c\": rpc error: code = NotFound desc = could not find container \"56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c\": container with ID starting with 56ebd76821a94b5a4ce527fc3be0acce75c5642d69edd0377632091217c8365c not found: ID does not exist"
Oct 31 15:37:39.868387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:39.868355    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:39.961991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:39.961960    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\" (UniqueName: \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\") pod \"d4708017-d175-486e-a29f-fcba166fa8ac\" (UID: \"d4708017-d175-486e-a29f-fcba166fa8ac\") "
Oct 31 15:37:39.964375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:39.964339    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69" pod "d4708017-d175-486e-a29f-fcba166fa8ac" (UID: "d4708017-d175-486e-a29f-fcba166fa8ac"). InnerVolumeSpecName "kube-api-access-9xh69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:40.063051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.063022    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xh69\" (UniqueName: \"kubernetes.io/projected/d4708017-d175-486e-a29f-fcba166fa8ac-kube-api-access-9xh69\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:40.579287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.579207    4549 generic.go:356] "Generic (PLEG): container finished" podID="d4708017-d175-486e-a29f-fcba166fa8ac" containerID="2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3" exitCode=137
Oct 31 15:37:40.579457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.579294    4549 scope.go:122] "RemoveContainer" containerID="2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3"
Oct 31 15:37:40.579457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.579302    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:40.591515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.591495    4549 scope.go:122] "RemoveContainer" containerID="2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3"
Oct 31 15:37:40.591907 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:40.591886    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3\": container with ID starting with 2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3 not found: ID does not exist" containerID="2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3"
Oct 31 15:37:40.591971 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:40.591919    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3"} err="failed to get container status \"2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3\": rpc error: code = NotFound desc = could not find container \"2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3\": container with ID starting with 2ebeeb87132e5ccfc58bedb4d1fb1d06a77542dd17d4fc01832a9d9314420cb3 not found: ID does not exist"
Oct 31 15:37:40.639643 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:40.639596    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7588-fe160e\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" pod="opendatahub/test-curl-7588-fe160e"
Oct 31 15:37:41.423299 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:41.423260    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" path="/var/lib/kubelet/pods/d4708017-d175-486e-a29f-fcba166fa8ac/volumes"
Oct 31 15:37:45.116801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.116712    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:37:45.117278 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.116904    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy" containerID="cri-o://a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f" gracePeriod=30
Oct 31 15:37:45.617966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.617928    4549 generic.go:356] "Generic (PLEG): container finished" podID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerID="227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b" exitCode=0
Oct 31 15:37:45.618184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.618011    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerDied","Data":"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"}
Oct 31 15:37:45.618508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.618477    4549 scope.go:122] "RemoveContainer" containerID="227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"
Oct 31 15:37:45.620205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.620172    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:37:45.620308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.620244    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:37:45.620308 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:45.620273    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:37:46.644034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.643991    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerStarted","Data":"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"}
Oct 31 15:37:46.667664 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.667621    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 15:37:46.668181 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.668167    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" containerName="test-curl-7588-fe160e"
Oct 31 15:37:46.668231 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.668183    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" containerName="test-curl-7588-fe160e"
Oct 31 15:37:46.668270 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.668261    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4708017-d175-486e-a29f-fcba166fa8ac" containerName="test-curl-7588-fe160e"
Oct 31 15:37:46.670327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.670311    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.735591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.724818    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c947d9-serving-cert\""
Oct 31 15:37:46.742050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.741854    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 15:37:46.747318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.747278    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.747498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.747366    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.747498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.747442    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.747498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.747480    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.850508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.849713    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.850508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.849774    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.850508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.849947    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.850508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.850006    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.853332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.853228    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.853332 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:46.853314    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-serving-cert: secret "maas-api-e2e-shared-b-w2-c947d9-serving-cert" not found
Oct 31 15:37:46.853501 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:46.853356    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls podName:c85fa0a0-4dcd-4e8b-9cf0-8653794958d8 nodeName:}" failed. No retries permitted until 2026-10-31 15:37:47.3533426 +0000 UTC m=+3108.079623556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" (UID: "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8") : secret "maas-api-e2e-shared-b-w2-c947d9-serving-cert" not found
Oct 31 15:37:46.854129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.854106    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:46.955802 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:46.955716    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:47.357248 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.357211    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:47.359749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.359724    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:47.582950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.582917    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:47.627645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.626909    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:37:47.631861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.630671    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 31 15:37:47.631861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.630731    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 31 15:37:47.992269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:47.992238    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 15:37:47.993813 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:37:47.993784    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85fa0a0_4dcd_4e8b_9cf0_8653794958d8.slice/crio-87f07c63772307ade991fc9893b8db61141952fc85e211a515b22e5c92d92e8b WatchSource:0}: Error finding container 87f07c63772307ade991fc9893b8db61141952fc85e211a515b22e5c92d92e8b: Status 404 returned error can't find the container with id 87f07c63772307ade991fc9893b8db61141952fc85e211a515b22e5c92d92e8b
Oct 31 15:37:48.621975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:48.621938    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 31 15:37:48.622249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:48.622221    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 31 15:37:48.659753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:48.659707    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" event={"ID":"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8","Type":"ContainerStarted","Data":"87f07c63772307ade991fc9893b8db61141952fc85e211a515b22e5c92d92e8b"}
Oct 31 15:37:48.812595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:48.811802    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:37:49.624595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:49.623969    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 31 15:37:49.624595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:49.624033    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 31 15:37:49.667784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:49.667752    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" event={"ID":"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8","Type":"ContainerStarted","Data":"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"}
Oct 31 15:37:49.667961 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:49.667887    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:37:49.792479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:49.792422    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" podStartSLOduration=2.929888851 podStartE2EDuration="3.792407659s" totalImagesPullingTime="862.518808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:37:49.791404257 +0000 UTC m=+3110.517685237" watchObservedRunningTime="2026-10-31 15:37:49.792407659 +0000 UTC m=+3110.518688637"
Oct 31 15:37:50.457625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.457598    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:37:50.489589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489539    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489614    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489649    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489686    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489729    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489752    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489780    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489785    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489806    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489863    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\") pod \"c04cb87e-0164-446c-824c-6740b2f8175f\" (UID: \"c04cb87e-0164-446c-824c-6740b2f8175f\") "
Oct 31 15:37:50.489909 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.489899    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490034    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490070    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490168    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490181    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490190    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.490288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490198    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.490612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.490368    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:37:50.492085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.492067    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:37:50.492296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.492279    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:50.492620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.492598    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "kube-api-access-wk6d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:37:50.492708 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.492630    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo" pod "c04cb87e-0164-446c-824c-6740b2f8175f" (UID: "c04cb87e-0164-446c-824c-6740b2f8175f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:37:50.591654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.591615    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04cb87e-0164-446c-824c-6740b2f8175f-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.591654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.591648    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04cb87e-0164-446c-824c-6740b2f8175f-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.591654 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.591668    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.591910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.591681    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6d6\" (UniqueName: \"kubernetes.io/projected/c04cb87e-0164-446c-824c-6740b2f8175f-kube-api-access-wk6d6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.591910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.591695    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04cb87e-0164-446c-824c-6740b2f8175f-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:37:50.625447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.624959    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 31 15:37:50.625447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.625033    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 31 15:37:50.675242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.675160    4549 generic.go:356] "Generic (PLEG): container finished" podID="c04cb87e-0164-446c-824c-6740b2f8175f" containerID="a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f" exitCode=0
Oct 31 15:37:50.675242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.675232    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"
Oct 31 15:37:50.675433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.675245    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" event={"ID":"c04cb87e-0164-446c-824c-6740b2f8175f","Type":"ContainerDied","Data":"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"}
Oct 31 15:37:50.675433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.675305    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8" event={"ID":"c04cb87e-0164-446c-824c-6740b2f8175f","Type":"ContainerDied","Data":"e2dc55e565861ce03d23cf5563fcb9406443ac1e6806fd134860ad272a3adb2e"}
Oct 31 15:37:50.675433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.675333    4549 scope.go:122] "RemoveContainer" containerID="a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"
Oct 31 15:37:50.686511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.686489    4549 scope.go:122] "RemoveContainer" containerID="a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"
Oct 31 15:37:50.686766 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:37:50.686746    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f\": container with ID starting with a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f not found: ID does not exist" containerID="a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"
Oct 31 15:37:50.686828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.686776    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f"} err="failed to get container status \"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f\": rpc error: code = NotFound desc = could not find container \"a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f\": container with ID starting with a9506e72ae90b7ee58e4056baa5192289940be5e20d765e2c38e76ad57222f9f not found: ID does not exist"
Oct 31 15:37:50.843947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.843901    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:37:50.847131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:50.847104    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-6m7z8"]
Oct 31 15:37:51.446766 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:51.446687    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" path="/var/lib/kubelet/pods/c04cb87e-0164-446c-824c-6740b2f8175f/volumes"
Oct 31 15:37:51.630079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:51.630029    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 31 15:37:51.630623 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:51.630102    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 31 15:37:52.626093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:52.626058    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:37:52.627377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:52.627354    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:37:55.681640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:37:55.681612    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 15:38:07.012329 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:38:07.012289    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:38:37.014068 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:38:37.014030    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:38:57.955045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:57.955009    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:38:57.955496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:57.955483    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy"
Oct 31 15:38:57.955545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:57.955497    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy"
Oct 31 15:38:57.955628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:57.955618    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04cb87e-0164-446c-824c-6740b2f8175f" containerName="istio-proxy"
Oct 31 15:38:57.959075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:57.959058    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.042037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.042007    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.042235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.042047    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.042235 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.042117    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.042366 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.042285    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rhg\" (UniqueName: \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.061005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.060968    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-982f43-serving-cert\""
Oct 31 15:38:58.064186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.064162    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:38:58.143290 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.143255    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.143484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.143308    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.143484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.143362    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.143616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.143530    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rhg\" (UniqueName: \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.143616 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:38:58.143534    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-serving-cert: secret "maas-api-e2e-extmod-a-982f43-serving-cert" not found
Oct 31 15:38:58.143743 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:38:58.143650    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls podName:39928868-76f2-45b5-b47c-ad8d4bd70ed8 nodeName:}" failed. No retries permitted until 2026-10-31 15:38:58.643626424 +0000 UTC m=+3179.369907384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls") pod "maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" (UID: "39928868-76f2-45b5-b47c-ad8d4bd70ed8") : secret "maas-api-e2e-extmod-a-982f43-serving-cert" not found
Oct 31 15:38:58.144430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.144404    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.146753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.146730    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.201468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.201436    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rhg\" (UniqueName: \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.648563 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.648527    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.651209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.651173    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-982f43-66856f94c5-96lss\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:58.872272 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:58.872232    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:38:59.247097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:38:59.247072    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:38:59.248158 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:38:59.248127    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39928868_76f2_45b5_b47c_ad8d4bd70ed8.slice/crio-7f2d7b190963475899bcf37333ab569555c50a65a7ce9aa50b5b92a07c259177 WatchSource:0}: Error finding container 7f2d7b190963475899bcf37333ab569555c50a65a7ce9aa50b5b92a07c259177: Status 404 returned error can't find the container with id 7f2d7b190963475899bcf37333ab569555c50a65a7ce9aa50b5b92a07c259177
Oct 31 15:39:00.165564 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:00.165536    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" event={"ID":"39928868-76f2-45b5-b47c-ad8d4bd70ed8","Type":"ContainerStarted","Data":"7f2d7b190963475899bcf37333ab569555c50a65a7ce9aa50b5b92a07c259177"}
Oct 31 15:39:01.175022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:01.174988    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" event={"ID":"39928868-76f2-45b5-b47c-ad8d4bd70ed8","Type":"ContainerStarted","Data":"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"}
Oct 31 15:39:01.175477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:01.175107    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:39:01.290493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:01.290437    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" podStartSLOduration=3.4124347520000002 podStartE2EDuration="4.290423657s" totalImagesPullingTime="877.988905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:39:01.288291954 +0000 UTC m=+3182.014572933" watchObservedRunningTime="2026-10-31 15:39:01.290423657 +0000 UTC m=+3182.016704635"
Oct 31 15:39:07.013377 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:39:07.013335    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:39:07.187034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:07.187004    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:39:08.963236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:08.963205    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:39:12.610620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.610514    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:39:12.615250 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.615225    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.669607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.669559    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:39:12.715443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.715408    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2cb1b7\"/\"openshift-service-ca.crt\""
Oct 31 15:39:12.715443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.715408    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2cb1b7\"/\"kube-root-ca.crt\""
Oct 31 15:39:12.716822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.716801    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2cb1b7\"/\"default-dockercfg-g5nhz\""
Oct 31 15:39:12.716822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.716806    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2cb1b7\"/\"explicit-model-2cb1b7-kserve-self-signed-certs\""
Oct 31 15:39:12.729599 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729553    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dvc\" (UniqueName: \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.729721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729631    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.729764 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729746    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.729801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729789    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.729938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729920    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.729975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.729961    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831014 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.830974    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831029    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831140    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dvc\" (UniqueName: \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831184    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831219    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831254    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831761 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831732    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831765    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.831891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.831766    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.834170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.834149    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.834288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.834254    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.889052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.888957    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dvc\" (UniqueName: \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\") pod \"explicit-model-2cb1b7-kserve-755698f64f-fxwjj\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") " pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:12.932595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:12.931277    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:13.316301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:13.316276    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:39:13.317553 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:39:13.317527    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod384d3d9f_ca61_4dda_b86c_961c2ed222aa.slice/crio-26a81516c984654e1b7f42beeca91ba1452e8bd63415d7c698bcc7fe9ad30c64 WatchSource:0}: Error finding container 26a81516c984654e1b7f42beeca91ba1452e8bd63415d7c698bcc7fe9ad30c64: Status 404 returned error can't find the container with id 26a81516c984654e1b7f42beeca91ba1452e8bd63415d7c698bcc7fe9ad30c64
Oct 31 15:39:14.267739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:14.267701    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" event={"ID":"384d3d9f-ca61-4dda-b86c-961c2ed222aa","Type":"ContainerStarted","Data":"006132bf504dd6ae2cc96c24957d644b41b27aeee96c426f40f302392533e60c"}
Oct 31 15:39:14.268206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:14.267753    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" event={"ID":"384d3d9f-ca61-4dda-b86c-961c2ed222aa","Type":"ContainerStarted","Data":"26a81516c984654e1b7f42beeca91ba1452e8bd63415d7c698bcc7fe9ad30c64"}
Oct 31 15:39:14.931974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:14.931941    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:22.932184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:22.932145    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:22.949783 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:22.949757    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:23.063165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:23.063105    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" podStartSLOduration=11.063091391 podStartE2EDuration="11.063091391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:39:14.382615152 +0000 UTC m=+3195.108896131" watchObservedRunningTime="2026-10-31 15:39:23.063091391 +0000 UTC m=+3203.789372415"
Oct 31 15:39:23.351136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:23.351109    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:39:23.986225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:23.986185    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" podUID="6a484924-7247-4344-8904-a7d68943c851" containerName="main" containerID="cri-o://3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8" gracePeriod=45
Oct 31 15:39:24.254129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.254066    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:39:24.345252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.345220    4549 generic.go:356] "Generic (PLEG): container finished" podID="6a484924-7247-4344-8904-a7d68943c851" containerID="3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8" exitCode=0
Oct 31 15:39:24.345466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.345263    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"
Oct 31 15:39:24.345466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.345311    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" event={"ID":"6a484924-7247-4344-8904-a7d68943c851","Type":"ContainerDied","Data":"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"}
Oct 31 15:39:24.345466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.345352    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8" event={"ID":"6a484924-7247-4344-8904-a7d68943c851","Type":"ContainerDied","Data":"004176aec98f0ed8ea964e6db97a4ffca2ab537b759166d0f6e5601b535e42ad"}
Oct 31 15:39:24.345466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.345375    4549 scope.go:122] "RemoveContainer" containerID="3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"
Oct 31 15:39:24.355502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.355474    4549 scope.go:122] "RemoveContainer" containerID="3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"
Oct 31 15:39:24.355920 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:39:24.355901    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8\": container with ID starting with 3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8 not found: ID does not exist" containerID="3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"
Oct 31 15:39:24.355999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.355931    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8"} err="failed to get container status \"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8\": rpc error: code = NotFound desc = could not find container \"3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8\": container with ID starting with 3a58b9197c1e119c4f7d450dfae73de479170410b0e359797cb6f6ad178101d8 not found: ID does not exist"
Oct 31 15:39:24.379421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379389    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.379616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379484    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.379616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379540    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.379804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379778    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\" (UniqueName: \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.379873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379809    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:24.379873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379860    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.379993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.379974    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\") pod \"6a484924-7247-4344-8904-a7d68943c851\" (UID: \"6a484924-7247-4344-8904-a7d68943c851\") "
Oct 31 15:39:24.380052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.380005    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:24.380589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.380477    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.380589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.380504    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.380939 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.380916    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:24.382034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.382013    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:39:24.382163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.382143    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:24.382406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.382386    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9" pod "6a484924-7247-4344-8904-a7d68943c851" (UID: "6a484924-7247-4344-8904-a7d68943c851"). InnerVolumeSpecName "kube-api-access-9znh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:39:24.481942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.481907    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.481942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.481941    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a484924-7247-4344-8904-a7d68943c851-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.482205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.481955    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a484924-7247-4344-8904-a7d68943c851-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.482205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.481971    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znh9\" (UniqueName: \"kubernetes.io/projected/6a484924-7247-4344-8904-a7d68943c851-kube-api-access-9znh9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:24.765722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.765688    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:39:24.768647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:24.768620    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/auth-test-model-aa864d-kserve-694cf5f5fc-h88n8"]
Oct 31 15:39:25.421182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:25.421147    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a484924-7247-4344-8904-a7d68943c851" path="/var/lib/kubelet/pods/6a484924-7247-4344-8904-a7d68943c851/volumes"
Oct 31 15:39:37.011109 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:39:37.011071    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:39:41.869957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:41.869922    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:39:41.870493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:41.870116    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" containerID="cri-o://5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e" gracePeriod=30
Oct 31 15:39:43.639230 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:43.639190    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-926mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:39:43.639782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:43.639255    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:39:46.028901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:46.028864    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:39:46.029756 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:46.029686    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy" containerID="cri-o://bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706" gracePeriod=30
Oct 31 15:39:47.132268 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.132241    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:39:47.153464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153406    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153470    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153497    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153527    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153631    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153656    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153688    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.153911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153801    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:47.153911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153813    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.154133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153953    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:47.154133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.153964    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\") pod \"05994620-9cca-47b4-b56b-a9b76bd824e2\" (UID: \"05994620-9cca-47b4-b56b-a9b76bd824e2\") "
Oct 31 15:39:47.154133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154098    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:47.154297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154175    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:39:47.154343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154328    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:47.154473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154452    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.154473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154473    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05994620-9cca-47b4-b56b-a9b76bd824e2-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.154799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154482    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.154799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154492    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.154799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.154501    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.156164 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.156137    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:39:47.156251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.156164    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:39:47.156373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.156357    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "kube-api-access-2n4mf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:39:47.156373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.156363    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy" pod "05994620-9cca-47b4-b56b-a9b76bd824e2" (UID: "05994620-9cca-47b4-b56b-a9b76bd824e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:47.256167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.256134    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.256167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.256167    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.256374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.256180    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4mf\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-kube-api-access-2n4mf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.256374 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.256194    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05994620-9cca-47b4-b56b-a9b76bd824e2-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:47.505178 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.505143    4549 generic.go:356] "Generic (PLEG): container finished" podID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerID="5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e" exitCode=0
Oct 31 15:39:47.505413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.505196    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"
Oct 31 15:39:47.505413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.505229    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" event={"ID":"05994620-9cca-47b4-b56b-a9b76bd824e2","Type":"ContainerDied","Data":"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"}
Oct 31 15:39:47.505413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.505299    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt" event={"ID":"05994620-9cca-47b4-b56b-a9b76bd824e2","Type":"ContainerDied","Data":"4d23c6e9e15668f8612d710b9cbe9d5c0075aa87b9a5a085e5254dfea2fb5186"}
Oct 31 15:39:47.505413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.505333    4549 scope.go:122] "RemoveContainer" containerID="5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"
Oct 31 15:39:47.515610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.515550    4549 scope.go:122] "RemoveContainer" containerID="5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"
Oct 31 15:39:47.515889 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:39:47.515871    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e\": container with ID starting with 5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e not found: ID does not exist" containerID="5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"
Oct 31 15:39:47.516214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.515899    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e"} err="failed to get container status \"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e\": rpc error: code = NotFound desc = could not find container \"5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e\": container with ID starting with 5f480d220e867082fa92cfe0718987ab244ef3402d4c718ff5279f735830df3e not found: ID does not exist"
Oct 31 15:39:47.672459 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.672421    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:39:47.675236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:47.675213    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-926mt"]
Oct 31 15:39:49.419737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:49.419703    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" path="/var/lib/kubelet/pods/05994620-9cca-47b4-b56b-a9b76bd824e2/volumes"
Oct 31 15:39:51.298156 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.298132    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:39:51.402956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.402870    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.402956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.402917    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.402970    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403109    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403191 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403160    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403191    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403244    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403290    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403366    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\") pod \"d2812e9f-67cc-4190-87e7-daed6ac347a4\" (UID: \"d2812e9f-67cc-4190-87e7-daed6ac347a4\") "
Oct 31 15:39:51.403494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403370    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:51.403616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403566    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:39:51.403744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403699    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:51.403876 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403854    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:51.405046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403973    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.405046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403996    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2812e9f-67cc-4190-87e7-daed6ac347a4-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.405046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.403926    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:51.405046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.404012    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.405046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.404133    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.405893 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.405868    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "kube-api-access-b96m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:39:51.406003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.405955    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:39:51.406113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.406096    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:39:51.406165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.406112    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo" pod "d2812e9f-67cc-4190-87e7-daed6ac347a4" (UID: "d2812e9f-67cc-4190-87e7-daed6ac347a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:39:51.505704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.505672    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.505704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.505701    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.505704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.505711    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.505932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.505721    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2812e9f-67cc-4190-87e7-daed6ac347a4-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.505932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.505730    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b96m8\" (UniqueName: \"kubernetes.io/projected/d2812e9f-67cc-4190-87e7-daed6ac347a4-kube-api-access-b96m8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:39:51.536923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.536894    4549 generic.go:356] "Generic (PLEG): container finished" podID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerID="bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706" exitCode=0
Oct 31 15:39:51.537101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.536956    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" event={"ID":"d2812e9f-67cc-4190-87e7-daed6ac347a4","Type":"ContainerDied","Data":"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"}
Oct 31 15:39:51.537101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.536958    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"
Oct 31 15:39:51.537101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.536999    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8" event={"ID":"d2812e9f-67cc-4190-87e7-daed6ac347a4","Type":"ContainerDied","Data":"32909758ba469cb6ce560ad3a5df82dbca82b5ed46973df1fe282d4ca172066b"}
Oct 31 15:39:51.537101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.537020    4549 scope.go:122] "RemoveContainer" containerID="bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"
Oct 31 15:39:51.549052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.549029    4549 scope.go:122] "RemoveContainer" containerID="bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"
Oct 31 15:39:51.549316 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:39:51.549295    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706\": container with ID starting with bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706 not found: ID does not exist" containerID="bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"
Oct 31 15:39:51.549390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.549328    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706"} err="failed to get container status \"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706\": rpc error: code = NotFound desc = could not find container \"bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706\": container with ID starting with bc205926307bd22487fd0f5f73a72dd91b529b0d6d7de4ffd16ae38582634706 not found: ID does not exist"
Oct 31 15:39:51.702388 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.702289    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:39:51.713780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:51.713725    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-rdzf8"]
Oct 31 15:39:53.419630 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:53.419596    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" path="/var/lib/kubelet/pods/d2812e9f-67cc-4190-87e7-daed6ac347a4/volumes"
Oct 31 15:39:57.431534 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:39:57.431496    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:40:04.074962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.074922    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:40:04.075697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075679    4549 cpu_manager.go:414] "containerMap: removing container" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy"
Oct 31 15:40:04.075750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075699    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy"
Oct 31 15:40:04.075750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075717    4549 cpu_manager.go:414] "containerMap: removing container" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy"
Oct 31 15:40:04.075750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075724    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy"
Oct 31 15:40:04.075842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075757    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6a484924-7247-4344-8904-a7d68943c851" containerName="main"
Oct 31 15:40:04.075842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075765    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a484924-7247-4344-8904-a7d68943c851" containerName="main"
Oct 31 15:40:04.075910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075893    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a484924-7247-4344-8904-a7d68943c851" containerName="main"
Oct 31 15:40:04.075910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075907    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2812e9f-67cc-4190-87e7-daed6ac347a4" containerName="istio-proxy"
Oct 31 15:40:04.075974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.075916    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="05994620-9cca-47b4-b56b-a9b76bd824e2" containerName="istio-proxy"
Oct 31 15:40:04.080449 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.080430    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.135460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.135423    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:40:04.181122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.181087    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-4d1877-serving-cert\""
Oct 31 15:40:04.267240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.267205    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.267456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.267258    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.267456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.267378    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.267590 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.267550    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njbt\" (UniqueName: \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.369040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.368952    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njbt\" (UniqueName: \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.369040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.368990    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.369040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.369014    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.369294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.369067    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.369819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.369797    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.371879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.371852    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.371988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.371873    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.425634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.425595    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njbt\" (UniqueName: \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\") pod \"maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:04.694496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:04.694389    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:05.069960 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:05.069936    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:40:05.071354 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:40:05.071321    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772ccb5a_21ba_49e1_a6b0_e1844a1357db.slice/crio-c29fbcd804bff9b406cfba364c2ca1b7f213f8c6c168387d0d96943ca7eaca77 WatchSource:0}: Error finding container c29fbcd804bff9b406cfba364c2ca1b7f213f8c6c168387d0d96943ca7eaca77: Status 404 returned error can't find the container with id c29fbcd804bff9b406cfba364c2ca1b7f213f8c6c168387d0d96943ca7eaca77
Oct 31 15:40:05.632883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:05.632846    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" event={"ID":"772ccb5a-21ba-49e1-a6b0-e1844a1357db","Type":"ContainerStarted","Data":"c29fbcd804bff9b406cfba364c2ca1b7f213f8c6c168387d0d96943ca7eaca77"}
Oct 31 15:40:06.642036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:06.641994    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" event={"ID":"772ccb5a-21ba-49e1-a6b0-e1844a1357db","Type":"ContainerStarted","Data":"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"}
Oct 31 15:40:06.642525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:06.642190    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:06.758099 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:06.758026    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" podStartSLOduration=1.854487384 podStartE2EDuration="2.758006508s" totalImagesPullingTime="903.519124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:06.756968735 +0000 UTC m=+3247.483249713" watchObservedRunningTime="2026-10-31 15:40:06.758006508 +0000 UTC m=+3247.484287487"
Oct 31 15:40:07.012599 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:40:07.012494    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:40:10.806852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.806817    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:40:10.811145 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.811124    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.837715 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.837683    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.837895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.837734    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.837953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.837901    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.837953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.837934    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.838061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.837962    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7zt\" (UniqueName: \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.838133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.838110    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.866752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.866724    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:40:10.911879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.911844    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24ca6e\"/\"openshift-service-ca.crt\""
Oct 31 15:40:10.912052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.911938    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-24ca6e\"/\"kube-root-ca.crt\""
Oct 31 15:40:10.912710 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.912683    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24ca6e\"/\"default-dockercfg-btqxs\""
Oct 31 15:40:10.912852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.912740    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-24ca6e\"/\"auth-test-model-24ca6e-kserve-self-signed-certs\""
Oct 31 15:40:10.938921 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.938882    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7zt\" (UniqueName: \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.939104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.938950    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.939104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.939061    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.939104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.939096    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.940296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.940269    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.940437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.940314    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.940804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.940780    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.941958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.941937    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.945528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.942035    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.946668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.946644    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.950006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.949982    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:10.997420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:10.997389    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7zt\" (UniqueName: \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\") pod \"auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") " pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:11.126906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:11.126808    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:11.511061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:11.511030    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:40:11.512171 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:40:11.512143    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba32872a_6de5_43bd_8813_e15a5e0a2b8e.slice/crio-5ff734f1792f99cf54bd1943921471ebabca789778f543d6b995540582585259 WatchSource:0}: Error finding container 5ff734f1792f99cf54bd1943921471ebabca789778f543d6b995540582585259: Status 404 returned error can't find the container with id 5ff734f1792f99cf54bd1943921471ebabca789778f543d6b995540582585259
Oct 31 15:40:11.675997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:11.675957    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" event={"ID":"ba32872a-6de5-43bd-8813-e15a5e0a2b8e","Type":"ContainerStarted","Data":"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"}
Oct 31 15:40:11.675997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:11.676006    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" event={"ID":"ba32872a-6de5-43bd-8813-e15a5e0a2b8e","Type":"ContainerStarted","Data":"5ff734f1792f99cf54bd1943921471ebabca789778f543d6b995540582585259"}
Oct 31 15:40:11.791885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:11.791832    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" podStartSLOduration=1.791817355 podStartE2EDuration="1.791817355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:11.788215444 +0000 UTC m=+3252.514496423" watchObservedRunningTime="2026-10-31 15:40:11.791817355 +0000 UTC m=+3252.518098333"
Oct 31 15:40:12.128018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.127939    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:12.454952 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.454826    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" podUID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" containerName="main" containerID="cri-o://006132bf504dd6ae2cc96c24957d644b41b27aeee96c426f40f302392533e60c" gracePeriod=45
Oct 31 15:40:12.656219 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.656188    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:40:12.685317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.685288    4549 generic.go:356] "Generic (PLEG): container finished" podID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" containerID="006132bf504dd6ae2cc96c24957d644b41b27aeee96c426f40f302392533e60c" exitCode=0
Oct 31 15:40:12.685504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.685365    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" event={"ID":"384d3d9f-ca61-4dda-b86c-961c2ed222aa","Type":"ContainerDied","Data":"006132bf504dd6ae2cc96c24957d644b41b27aeee96c426f40f302392533e60c"}
Oct 31 15:40:12.724417 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.724388    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:40:12.769765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.769728    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.769966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.769849    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.769966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.769893    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\" (UniqueName: \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.769966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.769943    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.770142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.770006    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:40:12.770142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.770011    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.770142 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.770111    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\") pod \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\" (UID: \"384d3d9f-ca61-4dda-b86c-961c2ed222aa\") "
Oct 31 15:40:12.770327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.770298    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:40:12.770425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.770399    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:40:12.771186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.771117    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:12.771186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.771145    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:12.771186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.771161    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:12.772378 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.772357    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:40:12.772826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.772810    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "kube-api-access-26dvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:40:12.773158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.773128    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm" pod "384d3d9f-ca61-4dda-b86c-961c2ed222aa" (UID: "384d3d9f-ca61-4dda-b86c-961c2ed222aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:40:12.872746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.872713    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/384d3d9f-ca61-4dda-b86c-961c2ed222aa-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:12.872746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.872742    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dvc\" (UniqueName: \"kubernetes.io/projected/384d3d9f-ca61-4dda-b86c-961c2ed222aa-kube-api-access-26dvc\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:12.872746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:12.872753    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/384d3d9f-ca61-4dda-b86c-961c2ed222aa-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:40:13.700151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:13.700042    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj" event={"ID":"384d3d9f-ca61-4dda-b86c-961c2ed222aa","Type":"ContainerDied","Data":"26a81516c984654e1b7f42beeca91ba1452e8bd63415d7c698bcc7fe9ad30c64"}
Oct 31 15:40:13.700151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:13.700104    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"
Oct 31 15:40:13.700151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:13.700119    4549 scope.go:122] "RemoveContainer" containerID="006132bf504dd6ae2cc96c24957d644b41b27aeee96c426f40f302392533e60c"
Oct 31 15:40:13.872011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:13.871972    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:40:13.921950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:13.921914    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2cb1b7/explicit-model-2cb1b7-kserve-755698f64f-fxwjj"]
Oct 31 15:40:15.419634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:15.419602    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" path="/var/lib/kubelet/pods/384d3d9f-ca61-4dda-b86c-961c2ed222aa/volumes"
Oct 31 15:40:21.127640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:21.127603    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:21.146291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:21.146261    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:21.770348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:21.770316    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:40:29.484700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.484647    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:40:29.485409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.485387    4549 cpu_manager.go:414] "containerMap: removing container" podUID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" containerName="main"
Oct 31 15:40:29.485454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.485414    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" containerName="main"
Oct 31 15:40:29.485632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.485617    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="384d3d9f-ca61-4dda-b86c-961c2ed222aa" containerName="main"
Oct 31 15:40:29.490773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.490745    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.580525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.580450    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-06ec34-openshift-default-dockercfg-27t97\""
Oct 31 15:40:29.605703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.605353    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681430    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681522    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681590    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqf8\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681624    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681660    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681691    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681723    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681770    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.687716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.681846    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.782958    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783002    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqf8\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783023    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783319    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783363    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783384    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783420    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783502    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783701 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783678    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783743    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783843 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783813    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.783936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.783871    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.785009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.784079    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.785009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.784550    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.786428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.786404    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.787107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.787090    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.840714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.840683    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:29.841022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:29.841000    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqf8\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\") pod \"e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") " pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:30.113728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.113629    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:30.543503 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.543467    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:40:30.548466 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:40:30.548433    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5466b706_8e1a_4b22_9600_8ca53fdf1b6e.slice/crio-061feb18b8f47f789df4465749c4db89daf0e118ed1aa35880677e28b42c1a03 WatchSource:0}: Error finding container 061feb18b8f47f789df4465749c4db89daf0e118ed1aa35880677e28b42c1a03: Status 404 returned error can't find the container with id 061feb18b8f47f789df4465749c4db89daf0e118ed1aa35880677e28b42c1a03
Oct 31 15:40:30.551315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.551273    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:40:30.551385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.551371    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:40:30.551422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.551413    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:40:30.824675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.824555    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerStarted","Data":"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"}
Oct 31 15:40:30.824675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.824651    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerStarted","Data":"061feb18b8f47f789df4465749c4db89daf0e118ed1aa35880677e28b42c1a03"}
Oct 31 15:40:30.943589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:30.943241    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podStartSLOduration=1.943220527 podStartE2EDuration="1.943220527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:30.939632159 +0000 UTC m=+3271.665913137" watchObservedRunningTime="2026-10-31 15:40:30.943220527 +0000 UTC m=+3271.669501506"
Oct 31 15:40:31.114515 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:31.114420    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:31.122591 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:31.122547    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:31.833363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:31.833330    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:31.834323 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:40:31.834302    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:40:37.012628 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:40:37.012530    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:41:01.274796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.274760    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:41:01.275320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.274955    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" podUID="f4b65214-e014-4629-8b23-a511285285a2" containerName="maas-api" containerID="cri-o://cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f" gracePeriod=30
Oct 31 15:41:01.570101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.570075    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:41:01.675855 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.675815    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\" (UniqueName: \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\") pod \"f4b65214-e014-4629-8b23-a511285285a2\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") "
Oct 31 15:41:01.676035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.675926    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\") pod \"f4b65214-e014-4629-8b23-a511285285a2\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") "
Oct 31 15:41:01.676035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.675988    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\") pod \"f4b65214-e014-4629-8b23-a511285285a2\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") "
Oct 31 15:41:01.676172 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.676155    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") pod \"f4b65214-e014-4629-8b23-a511285285a2\" (UID: \"f4b65214-e014-4629-8b23-a511285285a2\") "
Oct 31 15:41:01.676313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.676292    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca" pod "f4b65214-e014-4629-8b23-a511285285a2" (UID: "f4b65214-e014-4629-8b23-a511285285a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:41:01.676901 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.676878    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b65214-e014-4629-8b23-a511285285a2-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:01.678403 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.678377    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs" pod "f4b65214-e014-4629-8b23-a511285285a2" (UID: "f4b65214-e014-4629-8b23-a511285285a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:41:01.678498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.678432    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz" pod "f4b65214-e014-4629-8b23-a511285285a2" (UID: "f4b65214-e014-4629-8b23-a511285285a2"). InnerVolumeSpecName "kube-api-access-cr7dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:41:01.678498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.678453    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls" pod "f4b65214-e014-4629-8b23-a511285285a2" (UID: "f4b65214-e014-4629-8b23-a511285285a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:41:01.777929 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.777893    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:01.777929 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.777919    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr7dz\" (UniqueName: \"kubernetes.io/projected/f4b65214-e014-4629-8b23-a511285285a2-kube-api-access-cr7dz\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:01.777929 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:01.777933    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b65214-e014-4629-8b23-a511285285a2-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:02.041027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.040988    4549 generic.go:356] "Generic (PLEG): container finished" podID="f4b65214-e014-4629-8b23-a511285285a2" containerID="cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f" exitCode=0
Oct 31 15:41:02.041233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.041042    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"
Oct 31 15:41:02.041233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.041059    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" event={"ID":"f4b65214-e014-4629-8b23-a511285285a2","Type":"ContainerDied","Data":"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"}
Oct 31 15:41:02.041233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.041101    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw" event={"ID":"f4b65214-e014-4629-8b23-a511285285a2","Type":"ContainerDied","Data":"5c76570c1d4cbd3d3ffe8cae1f44c1cedf68173c2886d952db81ad40195c7ebf"}
Oct 31 15:41:02.041233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.041122    4549 scope.go:122] "RemoveContainer" containerID="cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"
Oct 31 15:41:02.051875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.051856    4549 scope.go:122] "RemoveContainer" containerID="cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"
Oct 31 15:41:02.052159 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:02.052136    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f\": container with ID starting with cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f not found: ID does not exist" containerID="cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"
Oct 31 15:41:02.052259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.052162    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f"} err="failed to get container status \"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f\": rpc error: code = NotFound desc = could not find container \"cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f\": container with ID starting with cc3356c7bf678a2254491b9e925ac02a6bfc38779a46e1933a8961069cae366f not found: ID does not exist"
Oct 31 15:41:02.235144 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.235096    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:41:02.239133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.239105    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2cb1b7-5fd89b9f7f-tk4tw"]
Oct 31 15:41:02.974279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:02.974253    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:41:03.124177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:03.124129    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:41:03.422718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:03.422677    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b65214-e014-4629-8b23-a511285285a2" path="/var/lib/kubelet/pods/f4b65214-e014-4629-8b23-a511285285a2/volumes"
Oct 31 15:41:05.746943 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:05.746906    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:41:05.747442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:05.747206    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" containerID="cri-o://3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a" gracePeriod=30
Oct 31 15:41:06.264013 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.263979    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-535e66"]
Oct 31 15:41:06.264462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.264450    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f4b65214-e014-4629-8b23-a511285285a2" containerName="maas-api"
Oct 31 15:41:06.264553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.264464    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b65214-e014-4629-8b23-a511285285a2" containerName="maas-api"
Oct 31 15:41:06.264610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.264563    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b65214-e014-4629-8b23-a511285285a2" containerName="maas-api"
Oct 31 15:41:06.269314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.269296    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:06.320905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.320871    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6nthq\""
Oct 31 15:41:06.322784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.322760    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-535e66"]
Oct 31 15:41:06.331613 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.331559    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtl8\" (UniqueName: \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\") pod \"test-curl-7579-535e66\" (UID: \"758400a3-1a1c-46c1-9460-6b49ba30f2b6\") " pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:06.433242 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.433206    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtl8\" (UniqueName: \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\") pod \"test-curl-7579-535e66\" (UID: \"758400a3-1a1c-46c1-9460-6b49ba30f2b6\") " pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:06.489478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.489449    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtl8\" (UniqueName: \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\") pod \"test-curl-7579-535e66\" (UID: \"758400a3-1a1c-46c1-9460-6b49ba30f2b6\") " pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:06.580885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.580857    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:06.956856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:06.956824    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-535e66"]
Oct 31 15:41:06.958911 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:41:06.958879    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod758400a3_1a1c_46c1_9460_6b49ba30f2b6.slice/crio-d2ac2f181cc959d72b837d13e217eb0c5e2eedb38ff8558c44100d22bebc7dab WatchSource:0}: Error finding container d2ac2f181cc959d72b837d13e217eb0c5e2eedb38ff8558c44100d22bebc7dab: Status 404 returned error can't find the container with id d2ac2f181cc959d72b837d13e217eb0c5e2eedb38ff8558c44100d22bebc7dab
Oct 31 15:41:07.016060 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:07.016019    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:41:07.077044 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:07.077009    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-535e66" event={"ID":"758400a3-1a1c-46c1-9460-6b49ba30f2b6","Type":"ContainerStarted","Data":"8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b"}
Oct 31 15:41:07.077227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:07.077059    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-535e66" event={"ID":"758400a3-1a1c-46c1-9460-6b49ba30f2b6","Type":"ContainerStarted","Data":"d2ac2f181cc959d72b837d13e217eb0c5e2eedb38ff8558c44100d22bebc7dab"}
Oct 31 15:41:07.191817 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:07.191754    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-535e66" podStartSLOduration=1.1917402209999999 podStartE2EDuration="1.191740221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:07.188280199 +0000 UTC m=+3307.914561178" watchObservedRunningTime="2026-10-31 15:41:07.191740221 +0000 UTC m=+3307.918021198"
Oct 31 15:41:07.628416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:07.628379    4549 patch_prober.go:29] interesting pod/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:41:07.628609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:07.628438    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:41:08.521465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:08.521420    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-535e66"]
Oct 31 15:41:08.525368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:08.525338    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-535e66"]
Oct 31 15:41:09.092476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:09.092432    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-535e66" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" containerName="test-curl-7579-535e66" containerID="cri-o://8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b" gracePeriod=2
Oct 31 15:41:09.143340 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:09.143298    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7579-535e66\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:09.466421 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:09.466331    4549 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7579-535e66\" is forbidden: User \"system:node:ip-10-0-139-115.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-115.us-west-1.compute.internal' and this object" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:10.840392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.840354    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:41:10.841178 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.841159    4549 cpu_manager.go:414] "containerMap: removing container" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" containerName="test-curl-7579-535e66"
Oct 31 15:41:10.841236 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.841182    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" containerName="test-curl-7579-535e66"
Oct 31 15:41:10.841318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.841306    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" containerName="test-curl-7579-535e66"
Oct 31 15:41:10.844995 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.844974    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.882947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.882907    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9p2w\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.882954    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883000    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883050    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883116    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883197    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883234    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883290    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.883506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.883360    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:10.897416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.897383    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5a0cef-openshift-default-dockercfg-7x5lt\""
Oct 31 15:41:10.956226 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:10.956171    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003287    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003389    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003425    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003477    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003543    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003599    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9p2w\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003628    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003683    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.003738    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.005455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.004462    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.009619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.007862    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.009619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.008468    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.009619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.008740    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.009619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.008993    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.026182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.026107    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.030026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.029892    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.061089 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.060960    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.063993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.063968    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9p2w\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\") pod \"e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") " pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.092357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.092325    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:41:11.108614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.108557    4549 generic.go:356] "Generic (PLEG): container finished" podID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerID="3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a" exitCode=0
Oct 31 15:41:11.108614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.108600    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerDied","Data":"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"}
Oct 31 15:41:11.108787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.108648    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"
Oct 31 15:41:11.108787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.108660    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh" event={"ID":"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0","Type":"ContainerDied","Data":"b68f91d5a34c48ea7f811a43e6bff2b2eb6cd13854984a73d85b684d6fc31bd0"}
Oct 31 15:41:11.108787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.108684    4549 scope.go:122] "RemoveContainer" containerID="3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"
Oct 31 15:41:11.151969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.151942    4549 scope.go:122] "RemoveContainer" containerID="227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"
Oct 31 15:41:11.161888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.161853    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:11.175714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.175686    4549 scope.go:122] "RemoveContainer" containerID="3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"
Oct 31 15:41:11.176117 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:11.175986    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a\": container with ID starting with 3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a not found: ID does not exist" containerID="3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"
Oct 31 15:41:11.176117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.176020    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a"} err="failed to get container status \"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a\": rpc error: code = NotFound desc = could not find container \"3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a\": container with ID starting with 3fc2ad88692ca843e740ee119b7b0c192e251401cfaa25a201f33bc832b8659a not found: ID does not exist"
Oct 31 15:41:11.176117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.176044    4549 scope.go:122] "RemoveContainer" containerID="227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"
Oct 31 15:41:11.176341 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:11.176319    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b\": container with ID starting with 227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b not found: ID does not exist" containerID="227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"
Oct 31 15:41:11.176397 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.176349    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b"} err="failed to get container status \"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b\": rpc error: code = NotFound desc = could not find container \"227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b\": container with ID starting with 227066f130c7c4f998a2eb8ea4528d3d3585f1e413b67a68d1730350edfe8d1b not found: ID does not exist"
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.204912    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205028    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205079    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205113    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205140    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205201    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205223    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205276    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.205317    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\") pod \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\" (UID: \"4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0\") "
Oct 31 15:41:11.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.206401    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:41:11.207134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.206660    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:41:11.207134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.206884    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:41:11.207134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.206936    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:41:11.207134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.206947    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:41:11.214528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.214473    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:41:11.214528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.214487    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:41:11.214890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.214855    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:41:11.216870 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.216839    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn" pod "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" (UID: "4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0"). InnerVolumeSpecName "kube-api-access-hjtcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:41:11.306758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306713    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.306758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306754    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.306758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306765    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306773    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306784    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306792    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306801    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjtcn\" (UniqueName: \"kubernetes.io/projected/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-kube-api-access-hjtcn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306811    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.307057 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.306819    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.359038 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.359003    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:11.408243 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.408201    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\" (UniqueName: \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\") pod \"758400a3-1a1c-46c1-9460-6b49ba30f2b6\" (UID: \"758400a3-1a1c-46c1-9460-6b49ba30f2b6\") "
Oct 31 15:41:11.410621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.410557    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8" pod "758400a3-1a1c-46c1-9460-6b49ba30f2b6" (UID: "758400a3-1a1c-46c1-9460-6b49ba30f2b6"). InnerVolumeSpecName "kube-api-access-kbtl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:41:11.421772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.421729    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" path="/var/lib/kubelet/pods/758400a3-1a1c-46c1-9460-6b49ba30f2b6/volumes"
Oct 31 15:41:11.509975 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.509938    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbtl8\" (UniqueName: \"kubernetes.io/projected/758400a3-1a1c-46c1-9460-6b49ba30f2b6-kube-api-access-kbtl8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:41:11.534364 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.534319    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:41:11.553188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.553092    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-2cb1b7-openshift-default-6c767f8445-ghpvh"]
Oct 31 15:41:11.617845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.617809    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:41:11.618093 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:41:11.617877    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e526a08_2fe3_4be7_be31_fe318b615dbf.slice/crio-e0e1802c1cfb7d48f2175a3cc234bf7e63732c7f3bb2dbb7d60d63eb69db6014 WatchSource:0}: Error finding container e0e1802c1cfb7d48f2175a3cc234bf7e63732c7f3bb2dbb7d60d63eb69db6014: Status 404 returned error can't find the container with id e0e1802c1cfb7d48f2175a3cc234bf7e63732c7f3bb2dbb7d60d63eb69db6014
Oct 31 15:41:11.621343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.621302    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:11.621427 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.621389    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:11.621478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:11.621430    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:12.119284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.119246    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerStarted","Data":"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"}
Oct 31 15:41:12.119842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.119310    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerStarted","Data":"e0e1802c1cfb7d48f2175a3cc234bf7e63732c7f3bb2dbb7d60d63eb69db6014"}
Oct 31 15:41:12.120790 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.120762    4549 generic.go:356] "Generic (PLEG): container finished" podID="758400a3-1a1c-46c1-9460-6b49ba30f2b6" containerID="8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b" exitCode=137
Oct 31 15:41:12.120930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.120830    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-535e66"
Oct 31 15:41:12.120930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.120838    4549 scope.go:122] "RemoveContainer" containerID="8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b"
Oct 31 15:41:12.130434 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.130410    4549 scope.go:122] "RemoveContainer" containerID="8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b"
Oct 31 15:41:12.130745 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:12.130720    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b\": container with ID starting with 8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b not found: ID does not exist" containerID="8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b"
Oct 31 15:41:12.130824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.130757    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b"} err="failed to get container status \"8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b\": rpc error: code = NotFound desc = could not find container \"8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b\": container with ID starting with 8144b083a66043505981f98f93ad6adab0453be2d3efe6ec693c8cf25d03531b not found: ID does not exist"
Oct 31 15:41:12.162644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.162620    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:12.168455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.168436    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:12.237862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:12.237815    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podStartSLOduration=2.237800845 podStartE2EDuration="2.237800845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:12.234271396 +0000 UTC m=+3312.960552375" watchObservedRunningTime="2026-10-31 15:41:12.237800845 +0000 UTC m=+3312.964081823"
Oct 31 15:41:13.130293 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:13.130258    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:13.131357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:13.131340    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:41:13.422030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:13.421951    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" path="/var/lib/kubelet/pods/4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0/volumes"
Oct 31 15:41:15.135042 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135008    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:41:15.135530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135515    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.135612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135532    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.135612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135545    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.135612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135551    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.135716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135652    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.135716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.135667    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3371c5-2f81-4bfc-bdd1-219bd5ab97b0" containerName="istio-proxy"
Oct 31 15:41:15.140437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.140416    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.242076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.242037    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c947d9\"/\"default-dockercfg-6zv62\""
Oct 31 15:41:15.242076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.242037    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-c947d9\"/\"auth-test-model-c947d9-kserve-self-signed-certs\""
Oct 31 15:41:15.242428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.242410    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c947d9\"/\"openshift-service-ca.crt\""
Oct 31 15:41:15.242545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.242524    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-c947d9\"/\"kube-root-ca.crt\""
Oct 31 15:41:15.245384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.245362    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:41:15.250438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250418    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.250517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250464    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.250517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250486    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.250657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250637    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwcv\" (UniqueName: \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.250697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250675    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.250879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.250861    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.351749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.351710    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.351749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.351750    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.351789    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwcv\" (UniqueName: \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.351823    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.351927    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352193 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.352013    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.352362    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352438 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.352405    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.352835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.352814    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.354273 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.354249    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.354401 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.354385    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.408861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.408785    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwcv\" (UniqueName: \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\") pod \"auth-test-model-c947d9-kserve-d5dbd9957-wjh2p\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") " pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.455413 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.455380    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:15.839504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:15.839475    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:41:15.841378 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:41:15.841355    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bc33145_c706_4b69_9bd7_59c90d21a509.slice/crio-90c7be92da381a32aefcbe40f182760ea41392358a7b2896a301f9dfd3b46047 WatchSource:0}: Error finding container 90c7be92da381a32aefcbe40f182760ea41392358a7b2896a301f9dfd3b46047: Status 404 returned error can't find the container with id 90c7be92da381a32aefcbe40f182760ea41392358a7b2896a301f9dfd3b46047
Oct 31 15:41:16.155324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:16.155223    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" event={"ID":"4bc33145-c706-4b69-9bd7-59c90d21a509","Type":"ContainerStarted","Data":"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"}
Oct 31 15:41:16.155324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:16.155267    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" event={"ID":"4bc33145-c706-4b69-9bd7-59c90d21a509","Type":"ContainerStarted","Data":"90c7be92da381a32aefcbe40f182760ea41392358a7b2896a301f9dfd3b46047"}
Oct 31 15:41:16.269998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:16.269942    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" podStartSLOduration=1.269927898 podStartE2EDuration="1.269927898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:16.268074174 +0000 UTC m=+3316.994355153" watchObservedRunningTime="2026-10-31 15:41:16.269927898 +0000 UTC m=+3316.996208941"
Oct 31 15:41:16.455895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:16.455815    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:25.455801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:25.455761    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:25.468920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:25.468895    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:26.242121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:26.242092    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:41:37.013558 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:37.013516    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:41:38.957422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:38.957383    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:41:38.962339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:38.962315    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.034869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.034832    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035043 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.034884    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rck\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035043 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035012    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035157    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035204    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035307 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035289    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035358 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035326    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035399    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.035488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.035475    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.062965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.062930    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:41:39.114595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.113910    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-69517a-openshift-default-dockercfg-hzz82\""
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138306    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138360    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138425    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138456    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138498    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138562    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138666    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138697    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rck\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.138740    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.139593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.139215    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.140298 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.139905    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.144595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.140479    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.144595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.140742    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.148932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.147601    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.154594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.149606    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.161832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.161673    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.197269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.197236    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.205232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.205197    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rck\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\") pod \"e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") " pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.277151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.277112    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:39.712067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.712033    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:41:39.713745 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:41:39.713722    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be39265_1513_426d_99e5_26ebd17d9951.slice/crio-1e5ea6526e947ff8065b6346cbf6a31e9e0ce33505339e9b5f52528d573a5649 WatchSource:0}: Error finding container 1e5ea6526e947ff8065b6346cbf6a31e9e0ce33505339e9b5f52528d573a5649: Status 404 returned error can't find the container with id 1e5ea6526e947ff8065b6346cbf6a31e9e0ce33505339e9b5f52528d573a5649
Oct 31 15:41:39.716986 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.716952    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:39.717093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.717043    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:39.717159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:39.717093    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:41:40.327744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:40.327704    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerStarted","Data":"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"}
Oct 31 15:41:40.328225 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:40.327759    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerStarted","Data":"1e5ea6526e947ff8065b6346cbf6a31e9e0ce33505339e9b5f52528d573a5649"}
Oct 31 15:41:40.443659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:40.443512    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podStartSLOduration=2.443493646 podStartE2EDuration="2.443493646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:40.442984876 +0000 UTC m=+3341.169265855" watchObservedRunningTime="2026-10-31 15:41:40.443493646 +0000 UTC m=+3341.169774627"
Oct 31 15:41:41.277749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:41.277712    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:41.284493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:41.284467    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:41.335633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:41.335604    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:41.336650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:41.336630    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:41:57.374696 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.374660    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:41:57.380533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.380514    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.434354 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.434327    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:41:57.472256 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.472219    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.472447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.472356    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.472447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.472404    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.472547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.472452    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvfz\" (UniqueName: \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.479665 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.479641    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-06ec34-serving-cert\""
Oct 31 15:41:57.574036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.574000    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.574036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.574043    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.574303 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:57.574153    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-serving-cert: secret "maas-api-e2e-extmod-b-06ec34-serving-cert" not found
Oct 31 15:41:57.574303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.574221    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvfz\" (UniqueName: \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.574303 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:41:57.574266    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls podName:275df403-66ca-442e-adf1-3f2eb3855bce nodeName:}" failed. No retries permitted until 2026-10-31 15:41:58.074238263 +0000 UTC m=+3358.800519219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls") pod "maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" (UID: "275df403-66ca-442e-adf1-3f2eb3855bce") : secret "maas-api-e2e-extmod-b-06ec34-serving-cert" not found
Oct 31 15:41:57.574483 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.574459    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.574771 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.574750    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.577138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.577116    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:57.633825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:57.633746    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvfz\" (UniqueName: \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:58.080065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:58.080022    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:58.082625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:58.082602    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:58.295796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:58.295758    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:41:58.674245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:58.674211    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:41:58.676207 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:41:58.676181    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275df403_66ca_442e_adf1_3f2eb3855bce.slice/crio-da8ff6eb4be973252ce87eba094ebe3d8b8f789c257894fde33e0b6ce4101536 WatchSource:0}: Error finding container da8ff6eb4be973252ce87eba094ebe3d8b8f789c257894fde33e0b6ce4101536: Status 404 returned error can't find the container with id da8ff6eb4be973252ce87eba094ebe3d8b8f789c257894fde33e0b6ce4101536
Oct 31 15:41:58.677501 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:58.677485    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:41:59.459457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:41:59.459416    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" event={"ID":"275df403-66ca-442e-adf1-3f2eb3855bce","Type":"ContainerStarted","Data":"da8ff6eb4be973252ce87eba094ebe3d8b8f789c257894fde33e0b6ce4101536"}
Oct 31 15:42:00.468263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:00.468228    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" event={"ID":"275df403-66ca-442e-adf1-3f2eb3855bce","Type":"ContainerStarted","Data":"b1fee762b6ef3c5b418378c1007e01d80f06b7ceb7d35999fcfd4f6949751233"}
Oct 31 15:42:00.468686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:00.468290    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:42:00.582678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:00.582624    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" podStartSLOduration=2.707351606 podStartE2EDuration="3.58260611s" totalImagesPullingTime="875.254504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:42:00.581069728 +0000 UTC m=+3361.307350707" watchObservedRunningTime="2026-10-31 15:42:00.58260611 +0000 UTC m=+3361.308887088"
Oct 31 15:42:06.485449 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:06.485419    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:42:07.011817 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:42:07.011776    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:42:32.875743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:32.875702    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:42:32.876185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:32.875765    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:42:34.703321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.703283    4549 generic.go:356] "Generic (PLEG): container finished" podID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerID="4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c" exitCode=0
Oct 31 15:42:34.703831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.703357    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerDied","Data":"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"}
Oct 31 15:42:34.703831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.703790    4549 scope.go:122] "RemoveContainer" containerID="4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"
Oct 31 15:42:34.705361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.705331    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:42:34.705454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.705400    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:42:34.705454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:34.705435    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:42:35.728728 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:35.728551    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerStarted","Data":"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"}
Oct 31 15:42:36.121595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:36.120739    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:42:36.121595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:36.120890    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 15:42:36.121595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:36.120944    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 15:42:37.039155 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:42:37.039114    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:42:37.128346 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:37.128001    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 15:42:37.128346 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:37.128069    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 15:42:38.118769 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:38.118735    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 15:42:38.119385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:38.119351    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 15:42:39.114947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:39.114910    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 15:42:39.115240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:39.115204    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 15:42:40.120006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:40.119961    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 31 15:42:40.120531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:40.120026    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 31 15:42:41.118860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:41.118832    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:42:41.119222 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:41.119200    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:42:41.119920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:42:41.119902    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:43:07.032036 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:43:07.032001    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:43:14.172735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.172694    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:43:14.173162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.172766    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:43:14.682301 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.682258    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:43:14.687382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.687352    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.705414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.705390    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.705550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.705432    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzll\" (UniqueName: \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.705652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.705624    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.705753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.705735    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.739264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.739231    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-69517a-serving-cert\""
Oct 31 15:43:14.742240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.742212    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:43:14.806931 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.806891    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.807134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.807028    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.807134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.807083    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzll\" (UniqueName: \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.807269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.807179    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.807269 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:43:14.807189    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-serving-cert: secret "maas-api-e2e-extmod-b-69517a-serving-cert" not found
Oct 31 15:43:14.807269 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:43:14.807259    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls podName:80ca2dc4-9fcf-4028-8354-9c0eb6922aae nodeName:}" failed. No retries permitted until 2026-10-31 15:43:15.307238831 +0000 UTC m=+3436.033519802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls") pod "maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" (UID: "80ca2dc4-9fcf-4028-8354-9c0eb6922aae") : secret "maas-api-e2e-extmod-b-69517a-serving-cert" not found
Oct 31 15:43:14.807718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.807698    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.810963 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.810924    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:14.864607 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:14.864549    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzll\" (UniqueName: \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:15.312704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:15.312621    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:15.315199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:15.315179    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69517a-54649b5995-x57nm\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:15.605048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:15.605001    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:15.978731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:15.978704    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:43:15.979713 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:43:15.979676    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ca2dc4_9fcf_4028_8354_9c0eb6922aae.slice/crio-d7d306e40fa82ce16330b0860246ea1212d88fcf55cfbd068eb4d6afa5a0dcee WatchSource:0}: Error finding container d7d306e40fa82ce16330b0860246ea1212d88fcf55cfbd068eb4d6afa5a0dcee: Status 404 returned error can't find the container with id d7d306e40fa82ce16330b0860246ea1212d88fcf55cfbd068eb4d6afa5a0dcee
Oct 31 15:43:16.010055 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.010024    4549 generic.go:356] "Generic (PLEG): container finished" podID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerID="251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e" exitCode=0
Oct 31 15:43:16.010213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.010098    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerDied","Data":"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"}
Oct 31 15:43:16.010585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.010548    4549 scope.go:122] "RemoveContainer" containerID="251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"
Oct 31 15:43:16.011504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.011486    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" event={"ID":"80ca2dc4-9fcf-4028-8354-9c0eb6922aae","Type":"ContainerStarted","Data":"d7d306e40fa82ce16330b0860246ea1212d88fcf55cfbd068eb4d6afa5a0dcee"}
Oct 31 15:43:16.012292 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.012260    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:16.012347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.012335    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:16.012386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:16.012372    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:17.031481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.031428    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerStarted","Data":"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"}
Oct 31 15:43:17.035790 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.035755    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" event={"ID":"80ca2dc4-9fcf-4028-8354-9c0eb6922aae","Type":"ContainerStarted","Data":"668c94d367d24afacc8e776dd1c26bd3acb6efbf4f8a7eebc045a50ac4265e31"}
Oct 31 15:43:17.035979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.035900    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:17.162399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.162319    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:43:17.162541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.162481    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:17.162541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.162521    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:17.262762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:17.262711    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" podStartSLOduration=2.391987043 podStartE2EDuration="3.262695585s" totalImagesPullingTime="870.708542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:43:17.260418989 +0000 UTC m=+3437.986699966" watchObservedRunningTime="2026-10-31 15:43:17.262695585 +0000 UTC m=+3437.988976602"
Oct 31 15:43:18.162341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:18.162306    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:18.162754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:18.162361    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:19.162601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:19.162546    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:19.162988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:19.162625    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:20.162478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:20.162438    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:20.162702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:20.162504    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:21.162860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:21.162827    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:21.163260 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:21.162882    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:22.163058 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:22.163021    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 31 15:43:22.163488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:22.163100    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 31 15:43:23.050694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:23.050669    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:43:23.167845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:23.167812    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:43:23.168262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:23.168173    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:43:23.169135 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:23.169112    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:43:37.013091 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:43:37.013052    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:43:57.377729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:57.377691    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:43:57.378140 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:57.377751    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:43:58.367493 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.367453    4549 generic.go:356] "Generic (PLEG): container finished" podID="0be39265-1513-426d-99e5-26ebd17d9951" containerID="d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7" exitCode=0
Oct 31 15:43:58.368119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.368086    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerDied","Data":"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"}
Oct 31 15:43:58.368746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.368723    4549 scope.go:122] "RemoveContainer" containerID="d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"
Oct 31 15:43:58.378367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.378319    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:58.378888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.378514    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:58.378888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:58.378668    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:43:59.378820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:43:59.378779    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerStarted","Data":"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"}
Oct 31 15:44:00.278333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:00.278297    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:44:00.278547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:00.278482    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 15:44:00.278651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:00.278533    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 15:44:01.278042 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:01.278008    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 15:44:01.278450 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:01.278066    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 15:44:02.278264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:02.278231    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 15:44:02.278712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:02.278281    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 15:44:03.277697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:03.277657    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 15:44:03.277889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:03.277715    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 15:44:04.277954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:04.277921    4549 patch_prober.go:29] interesting pod/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 31 15:44:04.278345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:04.277987    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 31 15:44:05.281852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:05.281824    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:44:05.282291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:05.282087    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:44:05.282861 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:05.282841    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:44:07.013752 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:44:07.013707    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:44:26.489610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.489550    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 15:44:26.493777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.493757    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.550437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.550409    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\""
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.601824    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.601957    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602039    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602178    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602223    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602250    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kpk\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602292    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602337    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.602370    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.608752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.604290    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 15:44:26.703998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.703966    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.703998 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704011    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704041    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kpk\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704194    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704269    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704303    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704345    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.704454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704407    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704468    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704669    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704870    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.704951    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.705001    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.705206 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.705100    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.706941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.706914    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.707112 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.707091    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.760787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.760715    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.760787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.760777    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kpk\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\") pod \"maas-default-gateway-openshift-default-67bf986f99-lbl5d\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:26.811352 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:26.811318    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:27.273176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.273142    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 15:44:27.274146 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:44:27.274118    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d90fda0_05d1_4eda_83d4_bd5a23d2879a.slice/crio-51ae8b85d4e093be4625a116d45a05c02e444066782d305595b63945d43933f9 WatchSource:0}: Error finding container 51ae8b85d4e093be4625a116d45a05c02e444066782d305595b63945d43933f9: Status 404 returned error can't find the container with id 51ae8b85d4e093be4625a116d45a05c02e444066782d305595b63945d43933f9
Oct 31 15:44:27.277136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.277098    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:44:27.277241 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.277191    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:44:27.277316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.277242    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:44:27.622699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.622608    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" event={"ID":"8d90fda0-05d1-4eda-83d4-bd5a23d2879a","Type":"ContainerStarted","Data":"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"}
Oct 31 15:44:27.622699 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.622664    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" event={"ID":"8d90fda0-05d1-4eda-83d4-bd5a23d2879a","Type":"ContainerStarted","Data":"51ae8b85d4e093be4625a116d45a05c02e444066782d305595b63945d43933f9"}
Oct 31 15:44:27.741541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.741478    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podStartSLOduration=1.741459727 podStartE2EDuration="1.741459727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:44:27.736441173 +0000 UTC m=+3508.462722174" watchObservedRunningTime="2026-10-31 15:44:27.741459727 +0000 UTC m=+3508.467740707"
Oct 31 15:44:27.811927 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:27.811889    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:28.820730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:28.820690    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:28.821347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:28.821314    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:29.816036 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:29.815992    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:29.816343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:29.816314    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:30.823024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:30.822977    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:30.823612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:30.823037    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:31.814990 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:31.814954    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:31.815279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:31.815250    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:32.816596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:32.811978    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:32.816596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:32.812045    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:33.812159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:33.812120    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:33.812375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:33.812194    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:34.811739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:34.811701    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Oct 31 15:44:34.812170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:34.811758    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Oct 31 15:44:36.852714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:36.852669    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:44:36.853228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:36.852735    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:44:37.015437 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:44:37.015394    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:44:37.093082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:37.093040    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:37.093526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:37.093501    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:37.094246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:37.094228    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 15:44:51.805608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:51.805551    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:44:51.807304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:51.807270    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" podUID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" containerName="maas-api" containerID="cri-o://f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6" gracePeriod=30
Oct 31 15:44:52.173418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.173381    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:44:52.275824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.275779    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\") pod \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") "
Oct 31 15:44:52.276029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.275939    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\" (UniqueName: \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\") pod \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") "
Oct 31 15:44:52.276029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.275990    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") pod \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") "
Oct 31 15:44:52.276152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.276026    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\") pod \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\" (UID: \"39928868-76f2-45b5-b47c-ad8d4bd70ed8\") "
Oct 31 15:44:52.276723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.276690    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca" pod "39928868-76f2-45b5-b47c-ad8d4bd70ed8" (UID: "39928868-76f2-45b5-b47c-ad8d4bd70ed8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:44:52.278881 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.278841    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs" pod "39928868-76f2-45b5-b47c-ad8d4bd70ed8" (UID: "39928868-76f2-45b5-b47c-ad8d4bd70ed8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:44:52.279174 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.279141    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg" pod "39928868-76f2-45b5-b47c-ad8d4bd70ed8" (UID: "39928868-76f2-45b5-b47c-ad8d4bd70ed8"). InnerVolumeSpecName "kube-api-access-t6rhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:44:52.279174 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.279149    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls" pod "39928868-76f2-45b5-b47c-ad8d4bd70ed8" (UID: "39928868-76f2-45b5-b47c-ad8d4bd70ed8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:44:52.378027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.377989    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rhg\" (UniqueName: \"kubernetes.io/projected/39928868-76f2-45b5-b47c-ad8d4bd70ed8-kube-api-access-t6rhg\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:44:52.378027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.378019    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:44:52.378027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.378030    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39928868-76f2-45b5-b47c-ad8d4bd70ed8-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:44:52.378267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.378038    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39928868-76f2-45b5-b47c-ad8d4bd70ed8-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:44:52.873598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.867738    4549 generic.go:356] "Generic (PLEG): container finished" podID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" containerID="f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6" exitCode=0
Oct 31 15:44:52.873598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.867865    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"
Oct 31 15:44:52.873598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.867971    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" event={"ID":"39928868-76f2-45b5-b47c-ad8d4bd70ed8","Type":"ContainerDied","Data":"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"}
Oct 31 15:44:52.873598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.868019    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss" event={"ID":"39928868-76f2-45b5-b47c-ad8d4bd70ed8","Type":"ContainerDied","Data":"7f2d7b190963475899bcf37333ab569555c50a65a7ce9aa50b5b92a07c259177"}
Oct 31 15:44:52.873598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.868050    4549 scope.go:122] "RemoveContainer" containerID="f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"
Oct 31 15:44:52.886824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.886777    4549 scope.go:122] "RemoveContainer" containerID="f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"
Oct 31 15:44:52.887188 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:44:52.887160    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6\": container with ID starting with f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6 not found: ID does not exist" containerID="f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"
Oct 31 15:44:52.887279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:52.887203    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6"} err="failed to get container status \"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6\": rpc error: code = NotFound desc = could not find container \"f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6\": container with ID starting with f37ed4c935964453cbf2db7b9b5a392a68d0533de302a42f779554a5133c8fd6 not found: ID does not exist"
Oct 31 15:44:53.034555 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:53.034508    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:44:53.039420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:53.039389    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-982f43-66856f94c5-96lss"]
Oct 31 15:44:53.420856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:44:53.420825    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" path="/var/lib/kubelet/pods/39928868-76f2-45b5-b47c-ad8d4bd70ed8/volumes"
Oct 31 15:45:00.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.216725    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:00.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.217600    4549 cpu_manager.go:414] "containerMap: removing container" podUID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" containerName="maas-api"
Oct 31 15:45:00.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.217620    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" containerName="maas-api"
Oct 31 15:45:00.218597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.217761    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="39928868-76f2-45b5-b47c-ad8d4bd70ed8" containerName="maas-api"
Oct 31 15:45:00.225383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.225048    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:00.226318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.225912    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.240030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.240002    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:00.254321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.251784    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.255148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.255108    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.268621 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.268014    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:00.271993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.271972    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.300141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300106    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsnq\" (UniqueName: \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.300338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300187    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkk97\" (UniqueName: \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.300338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300227    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprf9\" (UniqueName: \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.300338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300333    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.300518 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300373    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.300518 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.300407    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.338636 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.335841    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 15:45:00.370030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.369978    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:00.391040 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.391007    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:00.401179 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401139    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsnq\" (UniqueName: \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401197    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401231    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkk97\" (UniqueName: \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401252    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprf9\" (UniqueName: \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401365    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401396    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.401414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401417    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.401835 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.401453    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvgd\" (UniqueName: \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.402148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.402126    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.402258 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.402243    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.402720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.402675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.432113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.432059    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:00.459683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.459632    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:00.465463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.465372    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkk97\" (UniqueName: \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\") pod \"maas-api-key-cleanup-29891025-hnrqq\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.466966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.466932    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsnq\" (UniqueName: \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.467999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.467976    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprf9\" (UniqueName: \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.502429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.502335    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvgd\" (UniqueName: \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.502827 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.502802    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.503668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.503644    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.559350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.559312    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvgd\" (UniqueName: \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.570748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.570717    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:00.582486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.582456    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:00.591884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.591851    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:00.598763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.598714    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:00.985108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:00.985077    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:00.985553 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:00.985525    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada6ecbf_b400_4e13_a303_d90495326dc4.slice/crio-efd156a321b3e028759e8170c2fee323a771fa498da032112557f5d344783a63 WatchSource:0}: Error finding container efd156a321b3e028759e8170c2fee323a771fa498da032112557f5d344783a63: Status 404 returned error can't find the container with id efd156a321b3e028759e8170c2fee323a771fa498da032112557f5d344783a63
Oct 31 15:45:01.036360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.036329    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:01.039306 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:01.039273    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902d8bfe_3976_422c_99c1_6812ca4c8f3e.slice/crio-29bc5c8c5e107387b95d402107adc66454b71b946e8c35e232b310950540afbd WatchSource:0}: Error finding container 29bc5c8c5e107387b95d402107adc66454b71b946e8c35e232b310950540afbd: Status 404 returned error can't find the container with id 29bc5c8c5e107387b95d402107adc66454b71b946e8c35e232b310950540afbd
Oct 31 15:45:01.040559 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.040536    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:01.042473 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:01.042438    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5043bb_a19c_4079_aa4d_eb4e688ab98d.slice/crio-ba167a31f0c4680518e8fa15684e5b4b10890c31fc94c7b151d98d54c6a23c39 WatchSource:0}: Error finding container ba167a31f0c4680518e8fa15684e5b4b10890c31fc94c7b151d98d54c6a23c39: Status 404 returned error can't find the container with id ba167a31f0c4680518e8fa15684e5b4b10890c31fc94c7b151d98d54c6a23c39
Oct 31 15:45:01.086740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.086700    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:01.088127 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:01.088097    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c2cc7c_dd37_4bdb_a50f_0516410d7055.slice/crio-b6af3ab2492ac2756f32eac531a3aec88029d2170937b6475b1d2cbb9a46e180 WatchSource:0}: Error finding container b6af3ab2492ac2756f32eac531a3aec88029d2170937b6475b1d2cbb9a46e180: Status 404 returned error can't find the container with id b6af3ab2492ac2756f32eac531a3aec88029d2170937b6475b1d2cbb9a46e180
Oct 31 15:45:01.947268 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.947229    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerStarted","Data":"afdf99f94e696b4c7f4345f8ab865f387dee9ce86775eaaef81f468352079228"}
Oct 31 15:45:01.947766 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.947295    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerStarted","Data":"29bc5c8c5e107387b95d402107adc66454b71b946e8c35e232b310950540afbd"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.949408    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerStarted","Data":"8d9383577a7053509dfb05f74f02df18b1a3bc6aea534b85c35f3c45833d3fa8"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.949465    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerStarted","Data":"b6af3ab2492ac2756f32eac531a3aec88029d2170937b6475b1d2cbb9a46e180"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.951209    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerStarted","Data":"be9dfa64f303fb6f8c027fe4da41884e122c17386b6f1cdf21ff1242f00e7a56"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.951252    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerStarted","Data":"ba167a31f0c4680518e8fa15684e5b4b10890c31fc94c7b151d98d54c6a23c39"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.952964    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerStarted","Data":"ee83d980c63ddcd88c6610f6d6fec6213f54cb08333e724e41a446c4e2ac508b"}
Oct 31 15:45:01.953082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:01.953002    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerStarted","Data":"efd156a321b3e028759e8170c2fee323a771fa498da032112557f5d344783a63"}
Oct 31 15:45:02.063482 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.063426    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" podStartSLOduration=2.063410457 podStartE2EDuration="2.063410457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:02.061360829 +0000 UTC m=+3542.787641808" watchObservedRunningTime="2026-10-31 15:45:02.063410457 +0000 UTC m=+3542.789691435"
Oct 31 15:45:02.176478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.176421    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" podStartSLOduration=2.176407731 podStartE2EDuration="2.176407731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:02.172507613 +0000 UTC m=+3542.898788592" watchObservedRunningTime="2026-10-31 15:45:02.176407731 +0000 UTC m=+3542.902688708"
Oct 31 15:45:02.286296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.286237    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" podStartSLOduration=2.286219626 podStartE2EDuration="2.286219626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:02.28503754 +0000 UTC m=+3543.011318523" watchObservedRunningTime="2026-10-31 15:45:02.286219626 +0000 UTC m=+3543.012500604"
Oct 31 15:45:02.399489 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.399422    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" podStartSLOduration=2.399399713 podStartE2EDuration="2.399399713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:02.39624762 +0000 UTC m=+3543.122528591" watchObservedRunningTime="2026-10-31 15:45:02.399399713 +0000 UTC m=+3543.125680692"
Oct 31 15:45:02.962860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.962819    4549 generic.go:356] "Generic (PLEG): container finished" podID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerID="8d9383577a7053509dfb05f74f02df18b1a3bc6aea534b85c35f3c45833d3fa8" exitCode=60
Oct 31 15:45:02.963369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.962969    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerDied","Data":"8d9383577a7053509dfb05f74f02df18b1a3bc6aea534b85c35f3c45833d3fa8"}
Oct 31 15:45:02.963717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.963690    4549 scope.go:122] "RemoveContainer" containerID="8d9383577a7053509dfb05f74f02df18b1a3bc6aea534b85c35f3c45833d3fa8"
Oct 31 15:45:02.965372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.965345    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerID="be9dfa64f303fb6f8c027fe4da41884e122c17386b6f1cdf21ff1242f00e7a56" exitCode=60
Oct 31 15:45:02.965479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.965448    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerDied","Data":"be9dfa64f303fb6f8c027fe4da41884e122c17386b6f1cdf21ff1242f00e7a56"}
Oct 31 15:45:02.965694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.965668    4549 scope.go:122] "RemoveContainer" containerID="be9dfa64f303fb6f8c027fe4da41884e122c17386b6f1cdf21ff1242f00e7a56"
Oct 31 15:45:02.968462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.968436    4549 generic.go:356] "Generic (PLEG): container finished" podID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerID="ee83d980c63ddcd88c6610f6d6fec6213f54cb08333e724e41a446c4e2ac508b" exitCode=60
Oct 31 15:45:02.968632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.968608    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerDied","Data":"ee83d980c63ddcd88c6610f6d6fec6213f54cb08333e724e41a446c4e2ac508b"}
Oct 31 15:45:02.968822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.968758    4549 scope.go:122] "RemoveContainer" containerID="ee83d980c63ddcd88c6610f6d6fec6213f54cb08333e724e41a446c4e2ac508b"
Oct 31 15:45:02.970417 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.970394    4549 generic.go:356] "Generic (PLEG): container finished" podID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerID="afdf99f94e696b4c7f4345f8ab865f387dee9ce86775eaaef81f468352079228" exitCode=60
Oct 31 15:45:02.970513 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.970471    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerDied","Data":"afdf99f94e696b4c7f4345f8ab865f387dee9ce86775eaaef81f468352079228"}
Oct 31 15:45:02.970714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:02.970697    4549 scope.go:122] "RemoveContainer" containerID="afdf99f94e696b4c7f4345f8ab865f387dee9ce86775eaaef81f468352079228"
Oct 31 15:45:03.983370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:03.983279    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerStarted","Data":"eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"}
Oct 31 15:45:03.985313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:03.985287    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerStarted","Data":"dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"}
Oct 31 15:45:03.987171 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:03.987147    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerStarted","Data":"a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"}
Oct 31 15:45:03.988944 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:03.988923    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerStarted","Data":"cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"}
Oct 31 15:45:04.999141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:04.999104    4549 generic.go:356] "Generic (PLEG): container finished" podID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerID="dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe" exitCode=60
Oct 31 15:45:04.999686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:04.999235    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerDied","Data":"dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"}
Oct 31 15:45:04.999686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:04.999310    4549 scope.go:122] "RemoveContainer" containerID="8d9383577a7053509dfb05f74f02df18b1a3bc6aea534b85c35f3c45833d3fa8"
Oct 31 15:45:04.999813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:04.999684    4549 scope.go:122] "RemoveContainer" containerID="dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"
Oct 31 15:45:05.000072 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:05.000038    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7_odh-ai-gateway-infra(93c2cc7c-dd37-4bdb-a50f-0516410d7055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055"
Oct 31 15:45:05.001360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.001340    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerID="a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8" exitCode=60
Oct 31 15:45:05.001469 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.001426    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerDied","Data":"a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"}
Oct 31 15:45:05.001773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.001757    4549 scope.go:122] "RemoveContainer" containerID="a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"
Oct 31 15:45:05.002068 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:05.002039    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88_odh-ai-gateway-infra(bd5043bb-a19c-4079-aa4d-eb4e688ab98d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d"
Oct 31 15:45:05.003343 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.003320    4549 generic.go:356] "Generic (PLEG): container finished" podID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerID="cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a" exitCode=60
Oct 31 15:45:05.003439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.003403    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerDied","Data":"cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"}
Oct 31 15:45:05.003757 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.003738    4549 scope.go:122] "RemoveContainer" containerID="cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"
Oct 31 15:45:05.004013 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:05.003994    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891025-hnrqq_odh-ai-gateway-infra(ada6ecbf-b400-4e13-a303-d90495326dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4"
Oct 31 15:45:05.005782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.005761    4549 generic.go:356] "Generic (PLEG): container finished" podID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerID="eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300" exitCode=60
Oct 31 15:45:05.005877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.005827    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerDied","Data":"eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"}
Oct 31 15:45:05.006060 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.006046    4549 scope.go:122] "RemoveContainer" containerID="eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"
Oct 31 15:45:05.006287 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:05.006272    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558_odh-ai-gateway-infra(902d8bfe-3976-422c-99c1-6812ca4c8f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e"
Oct 31 15:45:05.012720 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.012701    4549 scope.go:122] "RemoveContainer" containerID="be9dfa64f303fb6f8c027fe4da41884e122c17386b6f1cdf21ff1242f00e7a56"
Oct 31 15:45:05.025805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.025783    4549 scope.go:122] "RemoveContainer" containerID="ee83d980c63ddcd88c6610f6d6fec6213f54cb08333e724e41a446c4e2ac508b"
Oct 31 15:45:05.038656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:05.038636    4549 scope.go:122] "RemoveContainer" containerID="afdf99f94e696b4c7f4345f8ab865f387dee9ce86775eaaef81f468352079228"
Oct 31 15:45:06.019284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:06.019256    4549 scope.go:122] "RemoveContainer" containerID="eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"
Oct 31 15:45:06.019908 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:06.019616    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558_odh-ai-gateway-infra(902d8bfe-3976-422c-99c1-6812ca4c8f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e"
Oct 31 15:45:06.021207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:06.021179    4549 scope.go:122] "RemoveContainer" containerID="dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"
Oct 31 15:45:06.021480 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:06.021458    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7_odh-ai-gateway-infra(93c2cc7c-dd37-4bdb-a50f-0516410d7055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055"
Oct 31 15:45:06.023274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:06.023251    4549 scope.go:122] "RemoveContainer" containerID="a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"
Oct 31 15:45:06.023502 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:06.023482    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88_odh-ai-gateway-infra(bd5043bb-a19c-4079-aa4d-eb4e688ab98d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d"
Oct 31 15:45:06.034543 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:06.034513    4549 scope.go:122] "RemoveContainer" containerID="cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"
Oct 31 15:45:06.034900 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:06.034873    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891025-hnrqq_odh-ai-gateway-infra(ada6ecbf-b400-4e13-a303-d90495326dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4"
Oct 31 15:45:07.013863 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:07.013814    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:45:11.094782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.094706    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:45:11.101148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.101123    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.156703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.156667    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:45:11.218063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.218035    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c947d9-openshift-default-dockercfg-g7j9f\""
Oct 31 15:45:11.240167 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240128    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240193    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240286    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240347    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240383    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240439    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240463    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240502    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.240803 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.240583    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5xn\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342175    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342220    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342270    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342304    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342336    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342370    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342393    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342437    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5xn\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342616    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342669    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.342845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342704    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.343020 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342924    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.343020 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.342965    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.343133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.343083    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.346048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.345884    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.346193 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.346147    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.399344 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.399315    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.399552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.399528    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5xn\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.415080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.414873    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:11.848155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.848125    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:45:11.848748 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:11.848722    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c708df_ee72_4192_8b59_0fe3ba98b852.slice/crio-34e4f160b1722b1d974a390e7174ab53f5d423658dcd29537894396944587256 WatchSource:0}: Error finding container 34e4f160b1722b1d974a390e7174ab53f5d423658dcd29537894396944587256: Status 404 returned error can't find the container with id 34e4f160b1722b1d974a390e7174ab53f5d423658dcd29537894396944587256
Oct 31 15:45:11.851377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.851343    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:11.851464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.851410    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:11.851464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.851442    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:11.887269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.887242    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:45:11.892957 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.892936    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.951365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.951327    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.951657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.951631    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.951791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.951779    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.951936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.951907    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.952157 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.952138    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpmr6\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.952295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.952278    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.952445 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.952427    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.952625 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.952604    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.952823 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.952803    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:11.968338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:11.968298    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:45:12.054431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054400    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054477    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054532    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054615    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054652    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054690    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054741    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpmr6\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054769    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.054996 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.054820    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.055155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.055003    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.055155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.055033    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.055254 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.055183    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.055305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.055270    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.055498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.055479    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.056972 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.056951    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.057473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.057456    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.079471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.079445    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" event={"ID":"a1c708df-ee72-4192-8b59-0fe3ba98b852","Type":"ContainerStarted","Data":"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"}
Oct 31 15:45:12.079619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.079495    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" event={"ID":"a1c708df-ee72-4192-8b59-0fe3ba98b852","Type":"ContainerStarted","Data":"34e4f160b1722b1d974a390e7174ab53f5d423658dcd29537894396944587256"}
Oct 31 15:45:12.111138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.111061    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.111487 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.111445    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpmr6\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\") pod \"maas-default-gateway-openshift-default-67bf986f99-s7xxq\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.206959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.206895    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podStartSLOduration=1.206876296 podStartE2EDuration="1.206876296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:12.205001605 +0000 UTC m=+3552.931282608" watchObservedRunningTime="2026-10-31 15:45:12.206876296 +0000 UTC m=+3552.933157275"
Oct 31 15:45:12.218666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.218633    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:12.416161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.416072    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:12.417691 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.417659    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:12.417908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.417722    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:12.657476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.657450    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:45:12.658251 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:12.658223    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56763e72_bc6c_491e_abbd_52dfce2d13dc.slice/crio-2c54f92dcc4ebc4a02136ebf08dd6cabe1faa3fdd6957edd594664c0140ec489 WatchSource:0}: Error finding container 2c54f92dcc4ebc4a02136ebf08dd6cabe1faa3fdd6957edd594664c0140ec489: Status 404 returned error can't find the container with id 2c54f92dcc4ebc4a02136ebf08dd6cabe1faa3fdd6957edd594664c0140ec489
Oct 31 15:45:12.661183 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.661148    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:12.661264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.661228    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:12.661264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:12.661256    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:13.090424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.090382    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" event={"ID":"56763e72-bc6c-491e-abbd-52dfce2d13dc","Type":"ContainerStarted","Data":"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"}
Oct 31 15:45:13.090626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.090434    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" event={"ID":"56763e72-bc6c-491e-abbd-52dfce2d13dc","Type":"ContainerStarted","Data":"2c54f92dcc4ebc4a02136ebf08dd6cabe1faa3fdd6957edd594664c0140ec489"}
Oct 31 15:45:13.219596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.212856    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podStartSLOduration=2.212835499 podStartE2EDuration="2.212835499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:13.205958807 +0000 UTC m=+3553.932239786" watchObservedRunningTime="2026-10-31 15:45:13.212835499 +0000 UTC m=+3553.939116502"
Oct 31 15:45:13.223809 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.223779    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:13.231033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.230679    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:13.231033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.230741    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:13.421663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.420960    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:13.421663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:13.421027    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:14.223597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:14.219226    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:14.223597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:14.219308    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:14.418997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:14.418957    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:14.419300 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:14.419271    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:15.226594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:15.226377    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:15.226594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:15.226448    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:15.418622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:15.418589    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:15.419043 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:15.419013    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:15.425200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:15.425176    4549 scope.go:122] "RemoveContainer" containerID="cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"
Oct 31 15:45:16.189506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.189469    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerStarted","Data":"7262c279ad6d21d350296471f8aef33aab8eadc4d00b65c08ed5fa0fafe3cc42"}
Oct 31 15:45:16.246651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.245686    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:16.246651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.245754    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:16.422596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.421005    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:16.422596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.421068    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:16.422596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:16.422222    4549 scope.go:122] "RemoveContainer" containerID="dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"
Oct 31 15:45:17.217804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.215259    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerStarted","Data":"9758c9b2123e07731949780df13c07308a292b5c6baa3387ddf93f6b1474c321"}
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.218886    4549 generic.go:356] "Generic (PLEG): container finished" podID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerID="7262c279ad6d21d350296471f8aef33aab8eadc4d00b65c08ed5fa0fafe3cc42" exitCode=60
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.218985    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerDied","Data":"7262c279ad6d21d350296471f8aef33aab8eadc4d00b65c08ed5fa0fafe3cc42"}
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.219036    4549 scope.go:122] "RemoveContainer" containerID="cdd2e7c418ec671d0404535dfdb2802a43f4ff6b1ec3e5a0bc4a291018c7fa7a"
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.219468    4549 scope.go:122] "RemoveContainer" containerID="7262c279ad6d21d350296471f8aef33aab8eadc4d00b65c08ed5fa0fafe3cc42"
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.219900    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:17.232610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.219946    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:17.256596 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:17.242851    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891025-hnrqq_odh-ai-gateway-infra(ada6ecbf-b400-4e13-a303-d90495326dc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4"
Oct 31 15:45:17.357755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.357630    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:17.444367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.443162    4549 scope.go:122] "RemoveContainer" containerID="eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"
Oct 31 15:45:17.444367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.443931    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:17.444367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:17.444002    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:18.227658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.225511    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:18.227658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.225581    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:18.241765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.241730    4549 generic.go:356] "Generic (PLEG): container finished" podID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerID="9758c9b2123e07731949780df13c07308a292b5c6baa3387ddf93f6b1474c321" exitCode=60
Oct 31 15:45:18.241956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.241759    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerDied","Data":"9758c9b2123e07731949780df13c07308a292b5c6baa3387ddf93f6b1474c321"}
Oct 31 15:45:18.241956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.241852    4549 scope.go:122] "RemoveContainer" containerID="dfe0602e98291ccb03e8e1bd810b5cbbce665751fe45e25d338f8dbced6544fe"
Oct 31 15:45:18.242265 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.242247    4549 scope.go:122] "RemoveContainer" containerID="9758c9b2123e07731949780df13c07308a292b5c6baa3387ddf93f6b1474c321"
Oct 31 15:45:18.242663 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:18.242635    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7_odh-ai-gateway-infra(93c2cc7c-dd37-4bdb-a50f-0516410d7055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055"
Oct 31 15:45:18.247139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.246900    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerStarted","Data":"0cb04194b6ddc0b0e8c3855ab46637c0940bcbed04c3a0bcd001ece0c740cea4"}
Oct 31 15:45:18.354321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.351034    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:18.432648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.432010    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 15:45:18.432648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.432069    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 15:45:18.433497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.433467    4549 scope.go:122] "RemoveContainer" containerID="a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"
Oct 31 15:45:18.452120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.452093    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:18.485003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.484975    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\") pod \"ada6ecbf-b400-4e13-a303-d90495326dc4\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") "
Oct 31 15:45:18.485176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.485147    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\" (UniqueName: \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\") pod \"ada6ecbf-b400-4e13-a303-d90495326dc4\" (UID: \"ada6ecbf-b400-4e13-a303-d90495326dc4\") "
Oct 31 15:45:18.485384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.485358    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca" pod "ada6ecbf-b400-4e13-a303-d90495326dc4" (UID: "ada6ecbf-b400-4e13-a303-d90495326dc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:18.485634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.485614    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ada6ecbf-b400-4e13-a303-d90495326dc4-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:18.487407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.487382    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97" pod "ada6ecbf-b400-4e13-a303-d90495326dc4" (UID: "ada6ecbf-b400-4e13-a303-d90495326dc4"). InnerVolumeSpecName "kube-api-access-mkk97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:18.587204 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:18.587170    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkk97\" (UniqueName: \"kubernetes.io/projected/ada6ecbf-b400-4e13-a303-d90495326dc4-kube-api-access-mkk97\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:19.219209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.219124    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:19.219209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.219184    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:19.256917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.256877    4549 generic.go:356] "Generic (PLEG): container finished" podID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerID="0cb04194b6ddc0b0e8c3855ab46637c0940bcbed04c3a0bcd001ece0c740cea4" exitCode=60
Oct 31 15:45:19.257109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.256918    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerDied","Data":"0cb04194b6ddc0b0e8c3855ab46637c0940bcbed04c3a0bcd001ece0c740cea4"}
Oct 31 15:45:19.257109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.256992    4549 scope.go:122] "RemoveContainer" containerID="eb85f3566dad3a0b0869b309f045a89991979e97704d8e8ec21ca85ecfbdd300"
Oct 31 15:45:19.257444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.257426    4549 scope.go:122] "RemoveContainer" containerID="0cb04194b6ddc0b0e8c3855ab46637c0940bcbed04c3a0bcd001ece0c740cea4"
Oct 31 15:45:19.257813 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:19.257787    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558_odh-ai-gateway-infra(902d8bfe-3976-422c-99c1-6812ca4c8f3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e"
Oct 31 15:45:19.261647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.261622    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerStarted","Data":"cf3b6442fa41f5bb8a8205889ee06b813fbe8005d245aa81de9e0901ac5ff866"}
Oct 31 15:45:19.263252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.263231    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq" event={"ID":"ada6ecbf-b400-4e13-a303-d90495326dc4","Type":"ContainerDied","Data":"efd156a321b3e028759e8170c2fee323a771fa498da032112557f5d344783a63"}
Oct 31 15:45:19.263252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.263250    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"
Oct 31 15:45:19.275390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.274628    4549 scope.go:122] "RemoveContainer" containerID="7262c279ad6d21d350296471f8aef33aab8eadc4d00b65c08ed5fa0fafe3cc42"
Oct 31 15:45:19.428981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.428952    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:19.429718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.429680    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:19.430489 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.430468    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:45:19.563923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.563900    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:19.587453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.587423    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:19.604129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.604080    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\") pod \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") "
Oct 31 15:45:19.604330 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.604228    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\" (UniqueName: \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\") pod \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\" (UID: \"93c2cc7c-dd37-4bdb-a50f-0516410d7055\") "
Oct 31 15:45:19.604877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.604656    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca" pod "93c2cc7c-dd37-4bdb-a50f-0516410d7055" (UID: "93c2cc7c-dd37-4bdb-a50f-0516410d7055"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:19.604877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.604796    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c2cc7c-dd37-4bdb-a50f-0516410d7055-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:19.606906 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.606756    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd" pod "93c2cc7c-dd37-4bdb-a50f-0516410d7055" (UID: "93c2cc7c-dd37-4bdb-a50f-0516410d7055"). InnerVolumeSpecName "kube-api-access-kzvgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:19.706370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.706339    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvgd\" (UniqueName: \"kubernetes.io/projected/93c2cc7c-dd37-4bdb-a50f-0516410d7055-kube-api-access-kzvgd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:19.715326 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.715289    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:19.719155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:19.719128    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891025-hnrqq"]
Oct 31 15:45:20.219216 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.219180    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:20.219395 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.219242    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:20.275152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.275128    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"
Oct 31 15:45:20.275298 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.275159    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7" event={"ID":"93c2cc7c-dd37-4bdb-a50f-0516410d7055","Type":"ContainerDied","Data":"b6af3ab2492ac2756f32eac531a3aec88029d2170937b6475b1d2cbb9a46e180"}
Oct 31 15:45:20.275373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.275316    4549 scope.go:122] "RemoveContainer" containerID="9758c9b2123e07731949780df13c07308a292b5c6baa3387ddf93f6b1474c321"
Oct 31 15:45:20.281300 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.281156    4549 generic.go:356] "Generic (PLEG): container finished" podID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerID="cf3b6442fa41f5bb8a8205889ee06b813fbe8005d245aa81de9e0901ac5ff866" exitCode=60
Oct 31 15:45:20.281300 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.281245    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerDied","Data":"cf3b6442fa41f5bb8a8205889ee06b813fbe8005d245aa81de9e0901ac5ff866"}
Oct 31 15:45:20.281937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.281915    4549 scope.go:122] "RemoveContainer" containerID="cf3b6442fa41f5bb8a8205889ee06b813fbe8005d245aa81de9e0901ac5ff866"
Oct 31 15:45:20.282258 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:20.282235    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88_odh-ai-gateway-infra(bd5043bb-a19c-4079-aa4d-eb4e688ab98d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d"
Oct 31 15:45:20.288738 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.288719    4549 scope.go:122] "RemoveContainer" containerID="a064b30fe82035f729680e54d15f5d6fe3aed7d4b4bd8285c67b53e76b9d49c8"
Oct 31 15:45:20.452776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.452746    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:20.498047 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.497955    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:20.502874 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.502843    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-aa864d-29891025-vbsz7"]
Oct 31 15:45:20.507161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.507137    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:20.517351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.517321    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\" (UniqueName: \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\") pod \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") "
Oct 31 15:45:20.517503 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.517377    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\") pod \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\" (UID: \"902d8bfe-3976-422c-99c1-6812ca4c8f3e\") "
Oct 31 15:45:20.517837 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.517808    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca" pod "902d8bfe-3976-422c-99c1-6812ca4c8f3e" (UID: "902d8bfe-3976-422c-99c1-6812ca4c8f3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:20.517983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.517918    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d8bfe-3976-422c-99c1-6812ca4c8f3e-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:20.520037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.520010    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq" pod "902d8bfe-3976-422c-99c1-6812ca4c8f3e" (UID: "902d8bfe-3976-422c-99c1-6812ca4c8f3e"). InnerVolumeSpecName "kube-api-access-mvsnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:20.619668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:20.619620    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvsnq\" (UniqueName: \"kubernetes.io/projected/902d8bfe-3976-422c-99c1-6812ca4c8f3e-kube-api-access-mvsnq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:21.219218 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.219188    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-s7xxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Oct 31 15:45:21.219379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.219240    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Oct 31 15:45:21.300514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.300486    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"
Oct 31 15:45:21.304077 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.304025    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558" event={"ID":"902d8bfe-3976-422c-99c1-6812ca4c8f3e","Type":"ContainerDied","Data":"29bc5c8c5e107387b95d402107adc66454b71b946e8c35e232b310950540afbd"}
Oct 31 15:45:21.304214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.304156    4549 scope.go:122] "RemoveContainer" containerID="0cb04194b6ddc0b0e8c3855ab46637c0940bcbed04c3a0bcd001ece0c740cea4"
Oct 31 15:45:21.424455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.424310    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" path="/var/lib/kubelet/pods/93c2cc7c-dd37-4bdb-a50f-0516410d7055/volumes"
Oct 31 15:45:21.425032 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.425005    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" path="/var/lib/kubelet/pods/ada6ecbf-b400-4e13-a303-d90495326dc4/volumes"
Oct 31 15:45:21.463228 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.463199    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:21.528425 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.528371    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:21.530497 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.530473    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\") pod \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") "
Oct 31 15:45:21.530759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.530741    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\" (UniqueName: \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\") pod \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\" (UID: \"bd5043bb-a19c-4079-aa4d-eb4e688ab98d\") "
Oct 31 15:45:21.530845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.530820    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca" pod "bd5043bb-a19c-4079-aa4d-eb4e688ab98d" (UID: "bd5043bb-a19c-4079-aa4d-eb4e688ab98d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:21.531237 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.531218    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:21.532309 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.532289    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2348-29891025-wz558"]
Oct 31 15:45:21.533788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.533765    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9" pod "bd5043bb-a19c-4079-aa4d-eb4e688ab98d" (UID: "bd5043bb-a19c-4079-aa4d-eb4e688ab98d"). InnerVolumeSpecName "kube-api-access-dprf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:21.633026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:21.632986    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dprf9\" (UniqueName: \"kubernetes.io/projected/bd5043bb-a19c-4079-aa4d-eb4e688ab98d-kube-api-access-dprf9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:22.309502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.309477    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"
Oct 31 15:45:22.309722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.309525    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88" event={"ID":"bd5043bb-a19c-4079-aa4d-eb4e688ab98d","Type":"ContainerDied","Data":"ba167a31f0c4680518e8fa15684e5b4b10890c31fc94c7b151d98d54c6a23c39"}
Oct 31 15:45:22.309722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.309625    4549 scope.go:122] "RemoveContainer" containerID="cf3b6442fa41f5bb8a8205889ee06b813fbe8005d245aa81de9e0901ac5ff866"
Oct 31 15:45:22.473913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.473875    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:22.477619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.477591    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891025-2rn88"]
Oct 31 15:45:22.744584 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.744488    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:22.744880 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.744856    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:22.745686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:22.745666    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:45:23.419229 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:23.419197    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" path="/var/lib/kubelet/pods/902d8bfe-3976-422c-99c1-6812ca4c8f3e/volumes"
Oct 31 15:45:23.419673 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:23.419659    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" path="/var/lib/kubelet/pods/bd5043bb-a19c-4079-aa4d-eb4e688ab98d/volumes"
Oct 31 15:45:26.993744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.993712    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 15:45:26.994261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994247    4549 cpu_manager.go:414] "containerMap: removing container" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994262    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994276    4549 cpu_manager.go:414] "containerMap: removing container" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994281    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994287    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994291    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994300    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994304    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994314    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994319    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994328    4549 cpu_manager.go:414] "containerMap: removing container" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994332    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994346    4549 cpu_manager.go:414] "containerMap: removing container" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994350    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994358    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994362    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994373    4549 cpu_manager.go:414] "containerMap: removing container" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994376    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994387    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994391    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994398    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994402    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994481    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994491    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994497    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994502    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5043bb-a19c-4079-aa4d-eb4e688ab98d" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994510    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994515    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994520    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994525    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994533    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada6ecbf-b400-4e13-a303-d90495326dc4" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994542    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:26.994644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.994553    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c2cc7c-dd37-4bdb-a50f-0516410d7055" containerName="cleanup"
Oct 31 15:45:26.999418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:26.999400    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103328    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103553 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103383    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppssk\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103592    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103654    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103701    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103736    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103765    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103810    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.103923 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.103863    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205140    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205210    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205258    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205295    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205325    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205372    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205422    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205486    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.205542    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppssk\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.206445    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.206722    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.206954    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.207158    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.208734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.207788    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.212922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.212896    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 15:45:27.213382 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.213361    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.221158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.221099    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.264514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.264445    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.264652 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.264554    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppssk\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\") pod \"maas-default-gateway-openshift-default-67bf986f99-mc5xl\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.316015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.315985    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:27.749327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.749303    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 15:45:27.749943 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:45:27.749916    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85568dbe_008f_4a0a_b567_a5e2a072d1dc.slice/crio-c5830dc0058dbda9b2c39425610d7ddb72e38e198fa048f17ab6542cbcf62601 WatchSource:0}: Error finding container c5830dc0058dbda9b2c39425610d7ddb72e38e198fa048f17ab6542cbcf62601: Status 404 returned error can't find the container with id c5830dc0058dbda9b2c39425610d7ddb72e38e198fa048f17ab6542cbcf62601
Oct 31 15:45:27.752618 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.752543    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:27.752697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.752680    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:27.752734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:27.752716    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:45:28.356629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:28.356596    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" event={"ID":"85568dbe-008f-4a0a-b567-a5e2a072d1dc","Type":"ContainerStarted","Data":"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"}
Oct 31 15:45:28.357041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:28.356656    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" event={"ID":"85568dbe-008f-4a0a-b567-a5e2a072d1dc","Type":"ContainerStarted","Data":"c5830dc0058dbda9b2c39425610d7ddb72e38e198fa048f17ab6542cbcf62601"}
Oct 31 15:45:28.471958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:28.471904    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podStartSLOduration=2.471888401 podStartE2EDuration="2.471888401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:28.471257905 +0000 UTC m=+3569.197538881" watchObservedRunningTime="2026-10-31 15:45:28.471888401 +0000 UTC m=+3569.198169379"
Oct 31 15:45:29.320593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:29.320285    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:29.328596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:29.327713    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:29.328596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:29.327780    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:30.318595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:30.317969    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:30.318595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:30.318034    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:31.319600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:31.318975    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:31.319600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:31.319036    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:32.320595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:32.319994    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:32.320595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:32.320054    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:33.320597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:33.320008    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:33.320597 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:33.320073    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:34.316639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:34.316589    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:34.316854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:34.316672    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:35.246488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:35.246453    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:45:35.316428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:35.316391    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 31 15:45:35.316646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:35.316457    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 31 15:45:37.028081 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:37.028043    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:45:37.317384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:37.317281    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:45:37.317384 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:37.317354    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:45:37.328987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:37.328952    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:37.330300 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:37.330278    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:37.331329 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:37.331308    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 15:45:49.181798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:49.181753    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:45:49.182269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:49.182025    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" containerID="cri-o://dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012" gracePeriod=30
Oct 31 15:45:50.268503 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.268459    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" podUID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" containerName="main" containerID="cri-o://4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e" gracePeriod=45
Oct 31 15:45:50.533498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.533444    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:45:50.557585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.557534    4549 generic.go:356] "Generic (PLEG): container finished" podID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" containerID="4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e" exitCode=0
Oct 31 15:45:50.557585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.557588    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"
Oct 31 15:45:50.557788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.557611    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" event={"ID":"ba32872a-6de5-43bd-8813-e15a5e0a2b8e","Type":"ContainerDied","Data":"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"}
Oct 31 15:45:50.557788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.557658    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk" event={"ID":"ba32872a-6de5-43bd-8813-e15a5e0a2b8e","Type":"ContainerDied","Data":"5ff734f1792f99cf54bd1943921471ebabca789778f543d6b995540582585259"}
Oct 31 15:45:50.557788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.557678    4549 scope.go:122] "RemoveContainer" containerID="4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"
Oct 31 15:45:50.568541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.568519    4549 scope.go:122] "RemoveContainer" containerID="4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"
Oct 31 15:45:50.568826 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:50.568809    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e\": container with ID starting with 4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e not found: ID does not exist" containerID="4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"
Oct 31 15:45:50.568875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.568833    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e"} err="failed to get container status \"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e\": rpc error: code = NotFound desc = could not find container \"4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e\": container with ID starting with 4b635dc7bf6ffde22e3570b3d26c3266b86fa1903eb9d801e440cebcb3b1703e not found: ID does not exist"
Oct 31 15:45:50.626850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.626817    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.626867    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.626912    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627005 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.626973    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\" (UniqueName: \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627059    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627100    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\") pod \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\" (UID: \"ba32872a-6de5-43bd-8813-e15a5e0a2b8e\") "
Oct 31 15:45:50.627284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627154    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:50.627284 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627204    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:50.627458 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627430    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:50.627681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627662    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.627759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627685    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.627759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.627694    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.629283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.629260    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:45:50.629797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.629775    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:50.629887 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.629868    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt" pod "ba32872a-6de5-43bd-8813-e15a5e0a2b8e" (UID: "ba32872a-6de5-43bd-8813-e15a5e0a2b8e"). InnerVolumeSpecName "kube-api-access-4f7zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:50.728798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.728759    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.728798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.728801    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.728994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.728821    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7zt\" (UniqueName: \"kubernetes.io/projected/ba32872a-6de5-43bd-8813-e15a5e0a2b8e-kube-api-access-4f7zt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:50.978302 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.978261    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:45:50.982377 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:50.982353    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-24ca6e/auth-test-model-24ca6e-kserve-6d8fdf795c-zhmpk"]
Oct 31 15:45:51.420105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:51.420078    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" path="/var/lib/kubelet/pods/ba32872a-6de5-43bd-8813-e15a5e0a2b8e/volumes"
Oct 31 15:45:53.170405 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:53.170366    4549 patch_prober.go:29] interesting pod/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:45:53.170915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:53.170434    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:45:54.466471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.466447    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:45:54.580195 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580160    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580195 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580195    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580221    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580245    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580342    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580453    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580489    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580521    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580537    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580624    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580655    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:54.580742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580664    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\") pod \"5e526a08-2fe3-4be7-be31-fe318b615dbf\" (UID: \"5e526a08-2fe3-4be7-be31-fe318b615dbf\") "
Oct 31 15:45:54.581039 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580814    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:54.581039 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.580922    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:54.581274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.581253    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.581381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.581349    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.581381 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.581367    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.581714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.581380    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.581714 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.581394    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e526a08-2fe3-4be7-be31-fe318b615dbf-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.582954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.582933    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:54.583299 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.583276    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "kube-api-access-t9p2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:54.583299 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.583286    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:45:54.583446 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.583380    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy" pod "5e526a08-2fe3-4be7-be31-fe318b615dbf" (UID: "5e526a08-2fe3-4be7-be31-fe318b615dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:45:54.588598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.588557    4549 generic.go:356] "Generic (PLEG): container finished" podID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerID="dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012" exitCode=0
Oct 31 15:45:54.588709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.588599    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerDied","Data":"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"}
Oct 31 15:45:54.588709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.588631    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"
Oct 31 15:45:54.588709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.588672    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n" event={"ID":"5e526a08-2fe3-4be7-be31-fe318b615dbf","Type":"ContainerDied","Data":"e0e1802c1cfb7d48f2175a3cc234bf7e63732c7f3bb2dbb7d60d63eb69db6014"}
Oct 31 15:45:54.588709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.588703    4549 scope.go:122] "RemoveContainer" containerID="dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"
Oct 31 15:45:54.616907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.616879    4549 scope.go:122] "RemoveContainer" containerID="251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"
Oct 31 15:45:54.640608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.639995    4549 scope.go:122] "RemoveContainer" containerID="dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"
Oct 31 15:45:54.641167 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:54.641137    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012\": container with ID starting with dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012 not found: ID does not exist" containerID="dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"
Oct 31 15:45:54.641309 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.641178    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012"} err="failed to get container status \"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012\": rpc error: code = NotFound desc = could not find container \"dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012\": container with ID starting with dcf7741964f77f05d5f3e3c1455c72928257d840b5440054a7fea1f2b5582012 not found: ID does not exist"
Oct 31 15:45:54.641309 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.641205    4549 scope.go:122] "RemoveContainer" containerID="251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"
Oct 31 15:45:54.641639 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:54.641615    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e\": container with ID starting with 251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e not found: ID does not exist" containerID="251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"
Oct 31 15:45:54.641779 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.641648    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e"} err="failed to get container status \"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e\": rpc error: code = NotFound desc = could not find container \"251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e\": container with ID starting with 251732f254f066f79e6f4ef6c77cb896178ae1e576b58214559b68694d28d50e not found: ID does not exist"
Oct 31 15:45:54.682964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.682923    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.682964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.682967    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9p2w\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-kube-api-access-t9p2w\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.683213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.682986    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.683213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.683000    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e526a08-2fe3-4be7-be31-fe318b615dbf-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:54.752414 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.752330    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:45:54.756985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:54.756959    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5a0cef-openshift-default-8465588cc7-8lg2n"]
Oct 31 15:45:55.423041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:55.423002    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" path="/var/lib/kubelet/pods/5e526a08-2fe3-4be7-be31-fe318b615dbf/volumes"
Oct 31 15:45:59.077287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.077250    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:45:59.077734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.077532    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" podUID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" containerName="maas-api" containerID="cri-o://439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e" gracePeriod=30
Oct 31 15:45:59.388432 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.388401    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:45:59.529842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.529807    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:45:59.530588 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530555    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.530634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530591    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.530634 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530627    4549 cpu_manager.go:414] "containerMap: removing container" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:59.530702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530635    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:59.530702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530659    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" containerName="main"
Oct 31 15:45:59.530702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530666    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" containerName="main"
Oct 31 15:45:59.530798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530704    4549 cpu_manager.go:414] "containerMap: removing container" podUID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" containerName="maas-api"
Oct 31 15:45:59.530798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530711    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" containerName="maas-api"
Oct 31 15:45:59.530798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530733    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.530798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530741    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.530920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530892    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d8bfe-3976-422c-99c1-6812ca4c8f3e" containerName="cleanup"
Oct 31 15:45:59.530920 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530909    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba32872a-6de5-43bd-8813-e15a5e0a2b8e" containerName="main"
Oct 31 15:45:59.530987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530924    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" containerName="maas-api"
Oct 31 15:45:59.530987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530934    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.530987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.530944    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e526a08-2fe3-4be7-be31-fe318b615dbf" containerName="istio-proxy"
Oct 31 15:45:59.534842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.534820    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.549165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.549134    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\") pod \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") "
Oct 31 15:45:59.549331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.549249    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\") pod \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") "
Oct 31 15:45:59.549420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.549403    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\") pod \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") "
Oct 31 15:45:59.549481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.549439    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\" (UniqueName: \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\") pod \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\" (UID: \"772ccb5a-21ba-49e1-a6b0-e1844a1357db\") "
Oct 31 15:45:59.549605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.549557    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca" pod "772ccb5a-21ba-49e1-a6b0-e1844a1357db" (UID: "772ccb5a-21ba-49e1-a6b0-e1844a1357db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:45:59.550118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.550093    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772ccb5a-21ba-49e1-a6b0-e1844a1357db-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:59.551993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.551962    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls" pod "772ccb5a-21ba-49e1-a6b0-e1844a1357db" (UID: "772ccb5a-21ba-49e1-a6b0-e1844a1357db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:45:59.556111 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.552783    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs" pod "772ccb5a-21ba-49e1-a6b0-e1844a1357db" (UID: "772ccb5a-21ba-49e1-a6b0-e1844a1357db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:45:59.556362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.556338    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt" pod "772ccb5a-21ba-49e1-a6b0-e1844a1357db" (UID: "772ccb5a-21ba-49e1-a6b0-e1844a1357db"). InnerVolumeSpecName "kube-api-access-9njbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:45:59.586351 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.586321    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-883697\""
Oct 31 15:45:59.586548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.586529    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-883697-dockercfg-ghzfr\""
Oct 31 15:45:59.588616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.588590    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.717748    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.717879    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l6gv\" (UniqueName: \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.718017    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.718033    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772ccb5a-21ba-49e1-a6b0-e1844a1357db-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.718047    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njbt\" (UniqueName: \"kubernetes.io/projected/772ccb5a-21ba-49e1-a6b0-e1844a1357db-kube-api-access-9njbt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.738621    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.745264    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.756959 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.753868    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:45:59.771539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.771087    4549 generic.go:356] "Generic (PLEG): container finished" podID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" containerID="439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e" exitCode=0
Oct 31 15:45:59.771539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.771224    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"
Oct 31 15:45:59.771539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.771324    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" event={"ID":"772ccb5a-21ba-49e1-a6b0-e1844a1357db","Type":"ContainerDied","Data":"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"}
Oct 31 15:45:59.771539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.771378    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv" event={"ID":"772ccb5a-21ba-49e1-a6b0-e1844a1357db","Type":"ContainerDied","Data":"c29fbcd804bff9b406cfba364c2ca1b7f213f8c6c168387d0d96943ca7eaca77"}
Oct 31 15:45:59.771539 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.771410    4549 scope.go:122] "RemoveContainer" containerID="439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"
Oct 31 15:45:59.787331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.787229    4549 scope.go:122] "RemoveContainer" containerID="439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"
Oct 31 15:45:59.787589 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:45:59.787549    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e\": container with ID starting with 439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e not found: ID does not exist" containerID="439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"
Oct 31 15:45:59.787695 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.787601    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e"} err="failed to get container status \"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e\": rpc error: code = NotFound desc = could not find container \"439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e\": container with ID starting with 439940c77eaff577e28570eb132dd3306af913ad0ef481a012d29e8de250c41e not found: ID does not exist"
Oct 31 15:45:59.819691 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.819656    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.819856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.819756    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l6gv\" (UniqueName: \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.819856 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.819828    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzjq\" (UniqueName: \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.819985 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.819872    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.820444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.820422    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.876078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.876046    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l6gv\" (UniqueName: \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\") pod \"payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:45:59.921256 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.921228    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzjq\" (UniqueName: \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.921424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.921274    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.922024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.922003    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.979185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.979148    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzjq\" (UniqueName: \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\") pod \"payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:45:59.979348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.979209    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:45:59.982238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:45:59.982215    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4d1877-845c96b94d-lw7nv"]
Oct 31 15:46:00.084594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.083882    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:46:00.154311 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.154281    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:46:00.517735 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:00.517708    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d00d0ce_58fa_445e_9b1a_83ea6414eacf.slice/crio-1dbaed2906036dabfa6464e11f7a598923cc3d66031e7c1114de520e48ca1469 WatchSource:0}: Error finding container 1dbaed2906036dabfa6464e11f7a598923cc3d66031e7c1114de520e48ca1469: Status 404 returned error can't find the container with id 1dbaed2906036dabfa6464e11f7a598923cc3d66031e7c1114de520e48ca1469
Oct 31 15:46:00.519002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.518971    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:46:00.560223 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.560198    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:46:00.561967 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:00.561799    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcec7ec_d23e_4d2c_b9f1_6a42a7091b5d.slice/crio-9c1e45367571e609ccf71999a7d85c5fe3182286db3f6c8b482af819b2a92ac5 WatchSource:0}: Error finding container 9c1e45367571e609ccf71999a7d85c5fe3182286db3f6c8b482af819b2a92ac5: Status 404 returned error can't find the container with id 9c1e45367571e609ccf71999a7d85c5fe3182286db3f6c8b482af819b2a92ac5
Oct 31 15:46:00.779905 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.779871    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" event={"ID":"0d00d0ce-58fa-445e-9b1a-83ea6414eacf","Type":"ContainerStarted","Data":"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"}
Oct 31 15:46:00.780111 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.779935    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:46:00.780111 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.779957    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" event={"ID":"0d00d0ce-58fa-445e-9b1a-83ea6414eacf","Type":"ContainerStarted","Data":"1dbaed2906036dabfa6464e11f7a598923cc3d66031e7c1114de520e48ca1469"}
Oct 31 15:46:00.781463 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.781440    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" event={"ID":"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d","Type":"ContainerStarted","Data":"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"}
Oct 31 15:46:00.781610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.781481    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" event={"ID":"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d","Type":"ContainerStarted","Data":"9c1e45367571e609ccf71999a7d85c5fe3182286db3f6c8b482af819b2a92ac5"}
Oct 31 15:46:00.781610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.781539    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:46:00.892972 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:00.892908    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" podStartSLOduration=1.892889568 podStartE2EDuration="1.892889568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:00.890785048 +0000 UTC m=+3601.617066027" watchObservedRunningTime="2026-10-31 15:46:00.892889568 +0000 UTC m=+3601.619170547"
Oct 31 15:46:01.009398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:01.009342    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" podStartSLOduration=2.009326806 podStartE2EDuration="2.009326806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:01.004103791 +0000 UTC m=+3601.730384773" watchObservedRunningTime="2026-10-31 15:46:01.009326806 +0000 UTC m=+3601.735607785"
Oct 31 15:46:01.420288 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:01.420251    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772ccb5a-21ba-49e1-a6b0-e1844a1357db" path="/var/lib/kubelet/pods/772ccb5a-21ba-49e1-a6b0-e1844a1357db/volumes"
Oct 31 15:46:03.028402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:03.028370    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:46:03.179118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:03.179094    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:46:06.793871 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.793835    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:46:06.799042 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.799017    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.884615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.884566    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-7d8d64-openshift-default-dockercfg-jrd8f\""
Oct 31 15:46:06.922584 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.922520    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.922925 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.922888    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4rw\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.923085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.923065    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.923262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.923246    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.923379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.923365    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.923504 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.923490    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.939431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.939401    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.939729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.939709    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.939889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.939874    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:06.966428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:06.961659    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043204    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043257    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043284    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043316    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043363    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4rw\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043383    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043427    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043444    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.043471    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.044670    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.044930    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.045153    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.045664    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.045913    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.046629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.046452    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.054132 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.054101    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.054275 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:46:07.054147    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:46:07.100589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.100542    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.101561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.101541    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4rw\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\") pod \"e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") " pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.116619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.116590    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:07.547105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.547073    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:46:07.547866 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:07.547840    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6654b8a6_e1a4_4b5c_92b2_7b727678f771.slice/crio-ba45b41bb13c8c016f8bbc4b54513664e5f29db0bf872e888e3d9eedfa578ba7 WatchSource:0}: Error finding container ba45b41bb13c8c016f8bbc4b54513664e5f29db0bf872e888e3d9eedfa578ba7: Status 404 returned error can't find the container with id ba45b41bb13c8c016f8bbc4b54513664e5f29db0bf872e888e3d9eedfa578ba7
Oct 31 15:46:07.550549 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.550518    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:46:07.550646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.550598    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:46:07.550646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.550627    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:46:07.843648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.843529    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" event={"ID":"6654b8a6-e1a4-4b5c-92b2-7b727678f771","Type":"ContainerStarted","Data":"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"}
Oct 31 15:46:07.843648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.843622    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" event={"ID":"6654b8a6-e1a4-4b5c-92b2-7b727678f771","Type":"ContainerStarted","Data":"ba45b41bb13c8c016f8bbc4b54513664e5f29db0bf872e888e3d9eedfa578ba7"}
Oct 31 15:46:07.960594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:07.960180    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" podStartSLOduration=1.9601654160000002 podStartE2EDuration="1.960165416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:07.957089705 +0000 UTC m=+3608.683370683" watchObservedRunningTime="2026-10-31 15:46:07.960165416 +0000 UTC m=+3608.686446394"
Oct 31 15:46:08.117363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:08.117268    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:08.123808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:08.123778    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:08.849922 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:08.849891    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:08.850828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:08.850803    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:46:11.791726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:11.791700    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:46:11.792129 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:11.791775    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:46:21.298333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:21.298298    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:46:24.839009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:24.838979    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:46:24.839526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:24.839185    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" podUID="275df403-66ca-442e-adf1-3f2eb3855bce" containerName="maas-api" containerID="cri-o://b1fee762b6ef3c5b418378c1007e01d80f06b7ceb7d35999fcfd4f6949751233" gracePeriod=30
Oct 31 15:46:24.969400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:24.969361    4549 generic.go:356] "Generic (PLEG): container finished" podID="275df403-66ca-442e-adf1-3f2eb3855bce" containerID="b1fee762b6ef3c5b418378c1007e01d80f06b7ceb7d35999fcfd4f6949751233" exitCode=0
Oct 31 15:46:24.969610 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:24.969533    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" event={"ID":"275df403-66ca-442e-adf1-3f2eb3855bce","Type":"ContainerDied","Data":"b1fee762b6ef3c5b418378c1007e01d80f06b7ceb7d35999fcfd4f6949751233"}
Oct 31 15:46:25.143303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.143278    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:46:25.192624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.192585    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\" (UniqueName: \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\") pod \"275df403-66ca-442e-adf1-3f2eb3855bce\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") "
Oct 31 15:46:25.192818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.192707    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") pod \"275df403-66ca-442e-adf1-3f2eb3855bce\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") "
Oct 31 15:46:25.192894 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.192844    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\") pod \"275df403-66ca-442e-adf1-3f2eb3855bce\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") "
Oct 31 15:46:25.192953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.192895    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\") pod \"275df403-66ca-442e-adf1-3f2eb3855bce\" (UID: \"275df403-66ca-442e-adf1-3f2eb3855bce\") "
Oct 31 15:46:25.193509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.193477    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca" pod "275df403-66ca-442e-adf1-3f2eb3855bce" (UID: "275df403-66ca-442e-adf1-3f2eb3855bce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:46:25.193666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.193645    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/275df403-66ca-442e-adf1-3f2eb3855bce-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:25.194999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.194967    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls" pod "275df403-66ca-442e-adf1-3f2eb3855bce" (UID: "275df403-66ca-442e-adf1-3f2eb3855bce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:46:25.195205 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.195181    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz" pod "275df403-66ca-442e-adf1-3f2eb3855bce" (UID: "275df403-66ca-442e-adf1-3f2eb3855bce"). InnerVolumeSpecName "kube-api-access-8kvfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:46:25.195833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.195806    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs" pod "275df403-66ca-442e-adf1-3f2eb3855bce" (UID: "275df403-66ca-442e-adf1-3f2eb3855bce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:46:25.295104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.295068    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:25.295104 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.295102    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvfz\" (UniqueName: \"kubernetes.io/projected/275df403-66ca-442e-adf1-3f2eb3855bce-kube-api-access-8kvfz\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:25.295359 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.295117    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/275df403-66ca-442e-adf1-3f2eb3855bce-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:25.977430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.977402    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"
Oct 31 15:46:25.977928 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.977425    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h" event={"ID":"275df403-66ca-442e-adf1-3f2eb3855bce","Type":"ContainerDied","Data":"da8ff6eb4be973252ce87eba094ebe3d8b8f789c257894fde33e0b6ce4101536"}
Oct 31 15:46:25.977928 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:25.977485    4549 scope.go:122] "RemoveContainer" containerID="b1fee762b6ef3c5b418378c1007e01d80f06b7ceb7d35999fcfd4f6949751233"
Oct 31 15:46:26.141554 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:26.141519    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:46:26.190845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:26.190808    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-06ec34-774cbd49cc-hps2h"]
Oct 31 15:46:27.420214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:27.420179    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275df403-66ca-442e-adf1-3f2eb3855bce" path="/var/lib/kubelet/pods/275df403-66ca-442e-adf1-3f2eb3855bce/volumes"
Oct 31 15:46:31.357083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:31.357042    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:46:31.357640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:31.357333    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy" containerID="cri-o://3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194" gracePeriod=30
Oct 31 15:46:36.076117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.076072    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:46:36.076726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.076711    4549 cpu_manager.go:414] "containerMap: removing container" podUID="275df403-66ca-442e-adf1-3f2eb3855bce" containerName="maas-api"
Oct 31 15:46:36.076781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.076727    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275df403-66ca-442e-adf1-3f2eb3855bce" containerName="maas-api"
Oct 31 15:46:36.076848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.076839    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="275df403-66ca-442e-adf1-3f2eb3855bce" containerName="maas-api"
Oct 31 15:46:36.080112 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.080096    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.106002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.105966    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.106859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.106831    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.107149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.107132    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.107334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.107316    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnl8n\" (UniqueName: \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.107533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.107517    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.107803 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.107786    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.152079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.152030    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7d8d64\"/\"kube-root-ca.crt\""
Oct 31 15:46:36.182380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.182345    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7d8d64\"/\"openshift-service-ca.crt\""
Oct 31 15:46:36.182560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.182501    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7d8d64\"/\"default-dockercfg-ckbcn\""
Oct 31 15:46:36.182672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.182646    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7d8d64\"/\"noten-model-7d8d64-kserve-self-signed-certs\""
Oct 31 15:46:36.185843 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.185823    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:46:36.208796 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.208767    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.208955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.208847    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.208955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.208880    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnl8n\" (UniqueName: \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.208955 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.208939    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.209148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.209007    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.209148 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.209090    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.209262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.209230    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.209587 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.209546    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.222138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.209564    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.222138 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.213741    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.222424 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.221629    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.265362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.265334    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnl8n\" (UniqueName: \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\") pod \"noten-model-7d8d64-kserve-658b64484f-ndcjp\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") " pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.321640 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.321596    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" podUID="4bc33145-c706-4b69-9bd7-59c90d21a509" containerName="main" containerID="cri-o://5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050" gracePeriod=45
Oct 31 15:46:36.392795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.392752    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:36.778508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.778476    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:46:36.787230 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.787209    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:46:36.788718 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.788692    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:46:36.788839 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:36.788763    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf4b551_a174_4488_aea2_8bb56597791a.slice/crio-df2f4f19cec02afca8d24e8e48c7901da4d0fc366b3974b91f7a4d445ba48b3d WatchSource:0}: Error finding container df2f4f19cec02afca8d24e8e48c7901da4d0fc366b3974b91f7a4d445ba48b3d: Status 404 returned error can't find the container with id df2f4f19cec02afca8d24e8e48c7901da4d0fc366b3974b91f7a4d445ba48b3d
Oct 31 15:46:36.829994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.829971    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830023    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830047    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830083    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830113    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\" (UniqueName: \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830144    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830172    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830209    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830236    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830234    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830259    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830307    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\") pod \"4bc33145-c706-4b69-9bd7-59c90d21a509\" (UID: \"4bc33145-c706-4b69-9bd7-59c90d21a509\") "
Oct 31 15:46:36.830752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830373    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830402    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830427    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.830752 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.830450    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\") pod \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\" (UID: \"5466b706-8e1a-4b22-9600-8ca53fdf1b6e\") "
Oct 31 15:46:36.832105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.831194    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.832105 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.831768    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.832877 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.832834    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.833154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.833128    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.833370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.833347    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.833729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.833697    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "kube-api-access-2dwcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:46:36.833816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.833731    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.834034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.834006    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:46:36.834150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.834070    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:46:36.834150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.834093    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:46:36.834283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.834151    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.834507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.834488    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.837162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.837138    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:46:36.837545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.837511    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm" pod "4bc33145-c706-4b69-9bd7-59c90d21a509" (UID: "4bc33145-c706-4b69-9bd7-59c90d21a509"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:46:36.838342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.838313    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8" pod "5466b706-8e1a-4b22-9600-8ca53fdf1b6e" (UID: "5466b706-8e1a-4b22-9600-8ca53fdf1b6e"). InnerVolumeSpecName "kube-api-access-zxqf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:46:36.933737 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.933556    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.934238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.933961    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.934238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934104    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc33145-c706-4b69-9bd7-59c90d21a509-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.934238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934127    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqf8\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-kube-api-access-zxqf8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.934238 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934143    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwcv\" (UniqueName: \"kubernetes.io/projected/4bc33145-c706-4b69-9bd7-59c90d21a509-kube-api-access-2dwcv\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934157    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934291    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934305    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934317    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934439    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934455    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc33145-c706-4b69-9bd7-59c90d21a509-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934471    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934486    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:36.935011 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:36.934610    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5466b706-8e1a-4b22-9600-8ca53fdf1b6e-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:46:37.027724 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:46:37.025191    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:46:37.065016 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.064980    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" event={"ID":"dcf4b551-a174-4488-aea2-8bb56597791a","Type":"ContainerStarted","Data":"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"}
Oct 31 15:46:37.065210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.065031    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" event={"ID":"dcf4b551-a174-4488-aea2-8bb56597791a","Type":"ContainerStarted","Data":"df2f4f19cec02afca8d24e8e48c7901da4d0fc366b3974b91f7a4d445ba48b3d"}
Oct 31 15:46:37.066467 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.066440    4549 generic.go:356] "Generic (PLEG): container finished" podID="4bc33145-c706-4b69-9bd7-59c90d21a509" containerID="5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050" exitCode=0
Oct 31 15:46:37.066626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.066471    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"
Oct 31 15:46:37.066626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.066525    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" event={"ID":"4bc33145-c706-4b69-9bd7-59c90d21a509","Type":"ContainerDied","Data":"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"}
Oct 31 15:46:37.066626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.066559    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p" event={"ID":"4bc33145-c706-4b69-9bd7-59c90d21a509","Type":"ContainerDied","Data":"90c7be92da381a32aefcbe40f182760ea41392358a7b2896a301f9dfd3b46047"}
Oct 31 15:46:37.066626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.066593    4549 scope.go:122] "RemoveContainer" containerID="5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"
Oct 31 15:46:37.068395 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.068377    4549 generic.go:356] "Generic (PLEG): container finished" podID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerID="3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194" exitCode=0
Oct 31 15:46:37.068500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.068420    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"
Oct 31 15:46:37.068500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.068436    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerDied","Data":"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"}
Oct 31 15:46:37.068500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.068492    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2" event={"ID":"5466b706-8e1a-4b22-9600-8ca53fdf1b6e","Type":"ContainerDied","Data":"061feb18b8f47f789df4465749c4db89daf0e118ed1aa35880677e28b42c1a03"}
Oct 31 15:46:37.077912 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.077762    4549 scope.go:122] "RemoveContainer" containerID="5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"
Oct 31 15:46:37.078271 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:46:37.078101    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050\": container with ID starting with 5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050 not found: ID does not exist" containerID="5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"
Oct 31 15:46:37.078271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.078134    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050"} err="failed to get container status \"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050\": rpc error: code = NotFound desc = could not find container \"5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050\": container with ID starting with 5e7213cb2ea73f908a11f9d2c1e84d3f6060f3786b8a0a876825fe568e420050 not found: ID does not exist"
Oct 31 15:46:37.078271 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.078157    4549 scope.go:122] "RemoveContainer" containerID="3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"
Oct 31 15:46:37.088034 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.088016    4549 scope.go:122] "RemoveContainer" containerID="4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"
Oct 31 15:46:37.097562 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.097547    4549 scope.go:122] "RemoveContainer" containerID="3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"
Oct 31 15:46:37.098467 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:46:37.098436    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194\": container with ID starting with 3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194 not found: ID does not exist" containerID="3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"
Oct 31 15:46:37.098588 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.098474    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194"} err="failed to get container status \"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194\": rpc error: code = NotFound desc = could not find container \"3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194\": container with ID starting with 3e8d37d60b6f33286549bae146c8e54f775680dea22c2725859c3ab3d2a61194 not found: ID does not exist"
Oct 31 15:46:37.098588 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.098498    4549 scope.go:122] "RemoveContainer" containerID="4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"
Oct 31 15:46:37.098823 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:46:37.098807    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c\": container with ID starting with 4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c not found: ID does not exist" containerID="4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"
Oct 31 15:46:37.098868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.098829    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c"} err="failed to get container status \"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c\": rpc error: code = NotFound desc = could not find container \"4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c\": container with ID starting with 4baaad9356533a867adae8a13bedbf75acdd035a76d1305773bc61ab3695cc3c not found: ID does not exist"
Oct 31 15:46:37.182410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.182357    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" podStartSLOduration=1.182342458 podStartE2EDuration="1.182342458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:37.180833261 +0000 UTC m=+3637.907114240" watchObservedRunningTime="2026-10-31 15:46:37.182342458 +0000 UTC m=+3637.908623437"
Oct 31 15:46:37.297704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.297665    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:46:37.303465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.303433    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-06ec34-openshift-default-86bbc95df-zztr2"]
Oct 31 15:46:37.393455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.393420    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:37.447670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.447635    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" path="/var/lib/kubelet/pods/5466b706-8e1a-4b22-9600-8ca53fdf1b6e/volumes"
Oct 31 15:46:37.449215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.449189    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:46:37.449421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:37.449408    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-c947d9/auth-test-model-c947d9-kserve-d5dbd9957-wjh2p"]
Oct 31 15:46:39.425500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:39.425464    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc33145-c706-4b69-9bd7-59c90d21a509" path="/var/lib/kubelet/pods/4bc33145-c706-4b69-9bd7-59c90d21a509/volumes"
Oct 31 15:46:40.752208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752100    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:46:40.752697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752681    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4bc33145-c706-4b69-9bd7-59c90d21a509" containerName="main"
Oct 31 15:46:40.752745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752699    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc33145-c706-4b69-9bd7-59c90d21a509" containerName="main"
Oct 31 15:46:40.752745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752712    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.752745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752717    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.752853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752817    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc33145-c706-4b69-9bd7-59c90d21a509" containerName="main"
Oct 31 15:46:40.752853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752825    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.752853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.752833    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.757692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.757670    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:40.812836 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.812803    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:46:40.861594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.860815    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-73b8d8\""
Oct 31 15:46:40.861594 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.861443    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-73b8d8-dockercfg-s9hvc\""
Oct 31 15:46:40.902930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.902833    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmfkx\" (UniqueName: \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:40.902930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.902886    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:40.917132 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.917094    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:46:40.917982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.917954    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.917982 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.917982    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5466b706-8e1a-4b22-9600-8ca53fdf1b6e" containerName="istio-proxy"
Oct 31 15:46:40.924875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.924850    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:40.974211 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:40.974164    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:46:41.004626 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.004508    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.004805 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.004641    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crsjw\" (UniqueName: \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.004895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.004797    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmfkx\" (UniqueName: \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:41.004895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.004842    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:41.005698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.005675    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:41.062077 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.062046    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmfkx\" (UniqueName: \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\") pod \"payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:41.072820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.072549    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:41.106850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.105812    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.106850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.105899    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crsjw\" (UniqueName: \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.106850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.106781    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.164165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.164115    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crsjw\" (UniqueName: \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\") pod \"payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.249945 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.249913    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:41.456110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.456081    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:46:41.456893 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:41.456867    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1911d49_ec44_451b_ad14_27ba00f49151.slice/crio-c17efc2de276a92efdfd1ea8fd251861fb24d32442457eadad79a2e225e56112 WatchSource:0}: Error finding container c17efc2de276a92efdfd1ea8fd251861fb24d32442457eadad79a2e225e56112: Status 404 returned error can't find the container with id c17efc2de276a92efdfd1ea8fd251861fb24d32442457eadad79a2e225e56112
Oct 31 15:46:41.625375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:41.625342    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:46:41.626137 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:46:41.626112    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719f3fc4_7f98_451d_93df_431cf106fb63.slice/crio-f4a5f3425cff9546409cbe2c3b3b9a2b62942f72a616702d253f861d4164c801 WatchSource:0}: Error finding container f4a5f3425cff9546409cbe2c3b3b9a2b62942f72a616702d253f861d4164c801: Status 404 returned error can't find the container with id f4a5f3425cff9546409cbe2c3b3b9a2b62942f72a616702d253f861d4164c801
Oct 31 15:46:42.108352 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.108314    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" event={"ID":"719f3fc4-7f98-451d-93df-431cf106fb63","Type":"ContainerStarted","Data":"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"}
Oct 31 15:46:42.108853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.108382    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:46:42.108853 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.108405    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" event={"ID":"719f3fc4-7f98-451d-93df-431cf106fb63","Type":"ContainerStarted","Data":"f4a5f3425cff9546409cbe2c3b3b9a2b62942f72a616702d253f861d4164c801"}
Oct 31 15:46:42.110059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.110038    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" event={"ID":"b1911d49-ec44-451b-ad14-27ba00f49151","Type":"ContainerStarted","Data":"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"}
Oct 31 15:46:42.110159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.110075    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" event={"ID":"b1911d49-ec44-451b-ad14-27ba00f49151","Type":"ContainerStarted","Data":"c17efc2de276a92efdfd1ea8fd251861fb24d32442457eadad79a2e225e56112"}
Oct 31 15:46:42.110159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.110121    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:42.221804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.221754    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" podStartSLOduration=2.221741406 podStartE2EDuration="2.221741406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:42.219148412 +0000 UTC m=+3642.945429384" watchObservedRunningTime="2026-10-31 15:46:42.221741406 +0000 UTC m=+3642.948022397"
Oct 31 15:46:42.332902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:42.332838    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" podStartSLOduration=2.332817289 podStartE2EDuration="2.332817289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:42.33045757 +0000 UTC m=+3643.056738549" watchObservedRunningTime="2026-10-31 15:46:42.332817289 +0000 UTC m=+3643.059098267"
Oct 31 15:46:46.393295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:46.393256    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:46.409522 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:46.409496    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:47.159399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:47.159372    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:46:53.120245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:53.120214    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:46:53.120644 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:46:53.120282    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:47:07.019727 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:07.019691    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:47:14.734118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.734080    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:47:14.739416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.739396    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.794645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.794615    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:47:14.813726 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.813692    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.813942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.813758    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.813942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.813828    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dtt\" (UniqueName: \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.813942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.813878    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.839549 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.839511    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-883697-serving-cert\""
Oct 31 15:47:14.915512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.915473    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.915721 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.915564    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.915721 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:14.915656    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-serving-cert: secret "maas-api-e2e-inf-a-883697-serving-cert" not found
Oct 31 15:47:14.915834 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:14.915752    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls podName:87244d9d-d1bd-4c60-a8b4-5719dc869b4a nodeName:}" failed. No retries permitted until 2026-10-31 15:47:15.415728257 +0000 UTC m=+3676.142009221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls") pod "maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" (UID: "87244d9d-d1bd-4c60-a8b4-5719dc869b4a") : secret "maas-api-e2e-inf-a-883697-serving-cert" not found
Oct 31 15:47:14.915834 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.915657    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dtt\" (UniqueName: \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.915958 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.915861    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.916177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.916158    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.919240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.919186    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:14.972675 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:14.972647    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dtt\" (UniqueName: \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:15.423052 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:15.423020    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:15.425558 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:15.425529    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") pod \"maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:15.656164 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:15.656128    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:16.031332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:16.031302    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:47:16.033584 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:47:16.033536    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87244d9d_d1bd_4c60_a8b4_5719dc869b4a.slice/crio-b1d5a7a2e296dd69e79d0c7429602a3288a034bb8c92a7cb65c152d12b870c0a WatchSource:0}: Error finding container b1d5a7a2e296dd69e79d0c7429602a3288a034bb8c92a7cb65c152d12b870c0a: Status 404 returned error can't find the container with id b1d5a7a2e296dd69e79d0c7429602a3288a034bb8c92a7cb65c152d12b870c0a
Oct 31 15:47:16.035150 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:16.035132    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:47:16.352518 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:16.352434    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" event={"ID":"87244d9d-d1bd-4c60-a8b4-5719dc869b4a","Type":"ContainerStarted","Data":"b1d5a7a2e296dd69e79d0c7429602a3288a034bb8c92a7cb65c152d12b870c0a"}
Oct 31 15:47:16.510547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:16.510507    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:47:16.511116 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:16.510806    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" podUID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" containerName="istio-proxy" containerID="cri-o://d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113" gracePeriod=30
Oct 31 15:47:17.361349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:17.361312    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" event={"ID":"87244d9d-d1bd-4c60-a8b4-5719dc869b4a","Type":"ContainerStarted","Data":"2c3639220cc9b9e0b1d7d942692cdc49dd3732e9f50e9c52381e7abdf9b8cf88"}
Oct 31 15:47:17.361836 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:17.361401    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:17.479277 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:17.477810    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" podStartSLOduration=2.58764252 podStartE2EDuration="3.477790019s" totalImagesPullingTime="890.147499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:47:17.474541873 +0000 UTC m=+3678.200822873" watchObservedRunningTime="2026-10-31 15:47:17.477790019 +0000 UTC m=+3678.204070998"
Oct 31 15:47:21.778841 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.778779    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:47:21.915844 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.915807    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916016 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.915909    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916016 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.915980    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916018    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916070    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916114    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916335 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916169    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916335 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916244    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916335 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916294    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\") pod \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\" (UID: \"6654b8a6-e1a4-4b5c-92b2-7b727678f771\") "
Oct 31 15:47:21.916557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916531    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:21.916653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916554    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:47:21.916751 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916728    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:21.916862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.916821    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:21.917085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.917008    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:21.917085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.917034    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:21.917085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.917048    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:21.917085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.917062    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:21.917369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.917095    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:21.919000 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.918975    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:21.919097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.919043    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "kube-api-access-qw4rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:21.919536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.919519    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:21.919669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.919650    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo" pod "6654b8a6-e1a4-4b5c-92b2-7b727678f771" (UID: "6654b8a6-e1a4-4b5c-92b2-7b727678f771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:47:21.919931 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:21.919909    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:47:22.017991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.017958    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:22.017991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.017990    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw4rw\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-kube-api-access-qw4rw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:22.018208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.018006    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6654b8a6-e1a4-4b5c-92b2-7b727678f771-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:22.018208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.018021    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:22.018208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.018033    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6654b8a6-e1a4-4b5c-92b2-7b727678f771-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:22.407949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.407913    4549 generic.go:356] "Generic (PLEG): container finished" podID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" containerID="d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113" exitCode=0
Oct 31 15:47:22.408153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.407989    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"
Oct 31 15:47:22.408153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.407997    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" event={"ID":"6654b8a6-e1a4-4b5c-92b2-7b727678f771","Type":"ContainerDied","Data":"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"}
Oct 31 15:47:22.408153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.408080    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv" event={"ID":"6654b8a6-e1a4-4b5c-92b2-7b727678f771","Type":"ContainerDied","Data":"ba45b41bb13c8c016f8bbc4b54513664e5f29db0bf872e888e3d9eedfa578ba7"}
Oct 31 15:47:22.408153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.408104    4549 scope.go:122] "RemoveContainer" containerID="d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"
Oct 31 15:47:22.420007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.419986    4549 scope.go:122] "RemoveContainer" containerID="d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"
Oct 31 15:47:22.420359 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:22.420338    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113\": container with ID starting with d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113 not found: ID does not exist" containerID="d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"
Oct 31 15:47:22.420426 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.420373    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113"} err="failed to get container status \"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113\": rpc error: code = NotFound desc = could not find container \"d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113\": container with ID starting with d5806eb4e3bb7ae6b7a96b4bff151731c7df65ae8e4d41ec2c2de14c5262e113 not found: ID does not exist"
Oct 31 15:47:22.572018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.571983    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:47:22.621055 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:22.621020    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-7d8d64-openshift-default-646f4598ff-xg2sv"]
Oct 31 15:47:23.377183 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:23.377157    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:47:23.420908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:23.420876    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" path="/var/lib/kubelet/pods/6654b8a6-e1a4-4b5c-92b2-7b727678f771/volumes"
Oct 31 15:47:33.232731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.232694    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:47:33.233122 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.232993    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" podUID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" containerName="maas-api" containerID="cri-o://668c94d367d24afacc8e776dd1c26bd3acb6efbf4f8a7eebc045a50ac4265e31" gracePeriod=30
Oct 31 15:47:33.504632 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.504519    4549 generic.go:356] "Generic (PLEG): container finished" podID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" containerID="668c94d367d24afacc8e776dd1c26bd3acb6efbf4f8a7eebc045a50ac4265e31" exitCode=0
Oct 31 15:47:33.504807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.504678    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" event={"ID":"80ca2dc4-9fcf-4028-8354-9c0eb6922aae","Type":"ContainerDied","Data":"668c94d367d24afacc8e776dd1c26bd3acb6efbf4f8a7eebc045a50ac4265e31"}
Oct 31 15:47:33.551045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.551016    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:47:33.698061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.695463    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") pod \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") "
Oct 31 15:47:33.698061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.695637    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\") pod \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") "
Oct 31 15:47:33.698061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.695699    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\" (UniqueName: \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\") pod \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") "
Oct 31 15:47:33.698061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.695778    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\") pod \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\" (UID: \"80ca2dc4-9fcf-4028-8354-9c0eb6922aae\") "
Oct 31 15:47:33.698061 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.696616    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca" pod "80ca2dc4-9fcf-4028-8354-9c0eb6922aae" (UID: "80ca2dc4-9fcf-4028-8354-9c0eb6922aae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:47:33.715244 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.715210    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs" pod "80ca2dc4-9fcf-4028-8354-9c0eb6922aae" (UID: "80ca2dc4-9fcf-4028-8354-9c0eb6922aae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:47:33.716775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.716746    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls" pod "80ca2dc4-9fcf-4028-8354-9c0eb6922aae" (UID: "80ca2dc4-9fcf-4028-8354-9c0eb6922aae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:47:33.719091 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.719063    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll" pod "80ca2dc4-9fcf-4028-8354-9c0eb6922aae" (UID: "80ca2dc4-9fcf-4028-8354-9c0eb6922aae"). InnerVolumeSpecName "kube-api-access-frzll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:33.798609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.798480    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:33.798609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.798520    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:33.798609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.798534    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frzll\" (UniqueName: \"kubernetes.io/projected/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-kube-api-access-frzll\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:33.798609 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:33.798548    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ca2dc4-9fcf-4028-8354-9c0eb6922aae-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:34.516900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:34.516873    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"
Oct 31 15:47:34.517477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:34.516901    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm" event={"ID":"80ca2dc4-9fcf-4028-8354-9c0eb6922aae","Type":"ContainerDied","Data":"d7d306e40fa82ce16330b0860246ea1212d88fcf55cfbd068eb4d6afa5a0dcee"}
Oct 31 15:47:34.517477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:34.516968    4549 scope.go:122] "RemoveContainer" containerID="668c94d367d24afacc8e776dd1c26bd3acb6efbf4f8a7eebc045a50ac4265e31"
Oct 31 15:47:34.679619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:34.679561    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:47:34.729465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:34.729425    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69517a-54649b5995-x57nm"]
Oct 31 15:47:35.422032 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:35.421994    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" path="/var/lib/kubelet/pods/80ca2dc4-9fcf-4028-8354-9c0eb6922aae/volumes"
Oct 31 15:47:36.860412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:36.860380    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:47:36.860917 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:36.860601    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy" containerID="cri-o://c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d" gracePeriod=30
Oct 31 15:47:36.942789 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:36.942727    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" containerName="main" containerID="cri-o://86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a" gracePeriod=45
Oct 31 15:47:37.068798 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:37.068735    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:47:37.221658 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.221626    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:47:37.352106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352075    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352291 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352234    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352329    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\" (UniqueName: \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352369    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352459    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352516    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\") pod \"dcf4b551-a174-4488-aea2-8bb56597791a\" (UID: \"dcf4b551-a174-4488-aea2-8bb56597791a\") "
Oct 31 15:47:37.352619 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352530    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:37.352795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352767    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:37.352999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352896    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:37.353263 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.352980    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.354748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.354724    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:37.354830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.354775    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:47:37.355003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.354989    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n" pod "dcf4b551-a174-4488-aea2-8bb56597791a" (UID: "dcf4b551-a174-4488-aea2-8bb56597791a"). InnerVolumeSpecName "kube-api-access-hnl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:37.454442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.454408    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.454442 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.454443    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.454702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.454457    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf4b551-a174-4488-aea2-8bb56597791a-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.454702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.454495    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnl8n\" (UniqueName: \"kubernetes.io/projected/dcf4b551-a174-4488-aea2-8bb56597791a-kube-api-access-hnl8n\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.454702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.454511    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf4b551-a174-4488-aea2-8bb56597791a-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:37.541677 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.541632    4549 generic.go:356] "Generic (PLEG): container finished" podID="dcf4b551-a174-4488-aea2-8bb56597791a" containerID="86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a" exitCode=0
Oct 31 15:47:37.541884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.541697    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"
Oct 31 15:47:37.541884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.541699    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" event={"ID":"dcf4b551-a174-4488-aea2-8bb56597791a","Type":"ContainerDied","Data":"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"}
Oct 31 15:47:37.541884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.541816    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" event={"ID":"dcf4b551-a174-4488-aea2-8bb56597791a","Type":"ContainerDied","Data":"df2f4f19cec02afca8d24e8e48c7901da4d0fc366b3974b91f7a4d445ba48b3d"}
Oct 31 15:47:37.541884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.541839    4549 scope.go:122] "RemoveContainer" containerID="86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"
Oct 31 15:47:37.552525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.552502    4549 scope.go:122] "RemoveContainer" containerID="86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"
Oct 31 15:47:37.552825 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:37.552800    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a\": container with ID starting with 86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a not found: ID does not exist" containerID="86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"
Oct 31 15:47:37.552913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.552834    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a"} err="failed to get container status \"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a\": rpc error: code = NotFound desc = could not find container \"86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a\": container with ID starting with 86af88ace4251d2a8168bfe48401c16e32a88e62ac8c2c26ef0ed3f44c1e256a not found: ID does not exist"
Oct 31 15:47:37.704155 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.704039    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:47:37.707994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:37.707966    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp"]
Oct 31 15:47:38.149896 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.149853    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-7d8d64/noten-model-7d8d64-kserve-658b64484f-ndcjp" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:47:38.553187 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553150    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:47:38.553704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553687    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" containerName="istio-proxy"
Oct 31 15:47:38.553704 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553702    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" containerName="istio-proxy"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553724    4549 cpu_manager.go:414] "containerMap: removing container" podUID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" containerName="maas-api"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553729    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" containerName="maas-api"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553737    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" containerName="main"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553742    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" containerName="main"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553836    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6654b8a6-e1a4-4b5c-92b2-7b727678f771" containerName="istio-proxy"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553846    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ca2dc4-9fcf-4028-8354-9c0eb6922aae" containerName="maas-api"
Oct 31 15:47:38.553915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.553852    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" containerName="main"
Oct 31 15:47:38.558729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.558708    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.623163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.623125    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:47:38.665785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.665752    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b21d8a-openshift-default-dockercfg-nzm2k\""
Oct 31 15:47:38.669406 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669375    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669511    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669600 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669563    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv95l\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669640    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669740 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669676    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669765    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669834    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669893    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.669974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.669923    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.771663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771619    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.771864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771682    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.771864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771749    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.771864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771833    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771873    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv95l\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771931    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.771979    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772064    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772152 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772130    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772166    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772181    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772227    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772519 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772497    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.772991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.772969    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.774653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.774626    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.774779 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.774762    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.828510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.828438    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.828510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.828482    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv95l\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\") pod \"e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") " pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:38.874969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:38.874930    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:39.334071 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:47:39.334040    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac093c8c_7308_4130_88dc_5eba33da8976.slice/crio-a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a WatchSource:0}: Error finding container a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a: Status 404 returned error can't find the container with id a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a
Oct 31 15:47:39.334416 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.334257    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:47:39.337106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.337076    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:47:39.337209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.337140    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:47:39.337209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.337168    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:47:39.420353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.420326    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf4b551-a174-4488-aea2-8bb56597791a" path="/var/lib/kubelet/pods/dcf4b551-a174-4488-aea2-8bb56597791a/volumes"
Oct 31 15:47:39.560514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.560469    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerStarted","Data":"6f434f65b55fe704969d3e8340c895bc70e59dc080a476dfab0be5cf8a19d476"}
Oct 31 15:47:39.560739 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.560543    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerStarted","Data":"a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a"}
Oct 31 15:47:39.679350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.679235    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podStartSLOduration=1.6792201599999999 podStartE2EDuration="1.67922016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:47:39.674591949 +0000 UTC m=+3700.400872927" watchObservedRunningTime="2026-10-31 15:47:39.67922016 +0000 UTC m=+3700.405501195"
Oct 31 15:47:39.876023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:39.875991    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:40.882321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:40.882293    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:41.573950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:41.573913    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:41.574949 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:41.574929    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:47:42.149836 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.149770    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:47:42.318118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318090    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318150    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318187    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318213    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318262    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318294    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318405    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318430    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318500    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318531    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318540    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:42.318612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318611    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\") pod \"0be39265-1513-426d-99e5-26ebd17d9951\" (UID: \"0be39265-1513-426d-99e5-26ebd17d9951\") "
Oct 31 15:47:42.318974 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.318947    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:47:42.319262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.319241    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0be39265-1513-426d-99e5-26ebd17d9951-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.319339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.319267    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.319339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.319282    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.319339 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.319296    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.319520 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.319359    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:42.320780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.320755    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:47:42.320902 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.320784    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:47:42.321063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.321044    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "kube-api-access-26rck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:42.321115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.321073    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token" pod "0be39265-1513-426d-99e5-26ebd17d9951" (UID: "0be39265-1513-426d-99e5-26ebd17d9951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:47:42.420054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.419972    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.420054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.420007    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0be39265-1513-426d-99e5-26ebd17d9951-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.420054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.420017    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0be39265-1513-426d-99e5-26ebd17d9951-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.420054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.420026    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rck\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-kube-api-access-26rck\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.420054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.420036    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0be39265-1513-426d-99e5-26ebd17d9951-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:47:42.582993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.582956    4549 generic.go:356] "Generic (PLEG): container finished" podID="0be39265-1513-426d-99e5-26ebd17d9951" containerID="c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d" exitCode=0
Oct 31 15:47:42.583165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.583008    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"
Oct 31 15:47:42.583165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.583032    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerDied","Data":"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"}
Oct 31 15:47:42.583165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.583089    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz" event={"ID":"0be39265-1513-426d-99e5-26ebd17d9951","Type":"ContainerDied","Data":"1e5ea6526e947ff8065b6346cbf6a31e9e0ce33505339e9b5f52528d573a5649"}
Oct 31 15:47:42.583165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.583111    4549 scope.go:122] "RemoveContainer" containerID="c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"
Oct 31 15:47:42.594473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.594458    4549 scope.go:122] "RemoveContainer" containerID="d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"
Oct 31 15:47:42.604094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.604074    4549 scope.go:122] "RemoveContainer" containerID="c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"
Oct 31 15:47:42.604349 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:42.604333    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d\": container with ID starting with c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d not found: ID does not exist" containerID="c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"
Oct 31 15:47:42.604400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.604360    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d"} err="failed to get container status \"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d\": rpc error: code = NotFound desc = could not find container \"c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d\": container with ID starting with c2bcc5d3e7398291941d9e1bff3fedaae3d9901a38deadad79b2a46815e0253d not found: ID does not exist"
Oct 31 15:47:42.604400 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.604376    4549 scope.go:122] "RemoveContainer" containerID="d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"
Oct 31 15:47:42.604615 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:47:42.604596    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7\": container with ID starting with d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7 not found: ID does not exist" containerID="d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"
Oct 31 15:47:42.604662 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.604622    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7"} err="failed to get container status \"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7\": rpc error: code = NotFound desc = could not find container \"d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7\": container with ID starting with d51316800a77f27f7f25270693d5dc54f91f78280582337b9fd10eb4ff5204b7 not found: ID does not exist"
Oct 31 15:47:42.748889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.748784    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:47:42.798509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:42.798477    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69517a-openshift-default-765f7ff4cc-ndfgz"]
Oct 31 15:47:43.419208 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:43.419175    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be39265-1513-426d-99e5-26ebd17d9951" path="/var/lib/kubelet/pods/0be39265-1513-426d-99e5-26ebd17d9951/volumes"
Oct 31 15:47:46.773511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.773480    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:47:46.774078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774063    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.774134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774080    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.774134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774092    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.774134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774097    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.774234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774193    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.774234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.774200    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be39265-1513-426d-99e5-26ebd17d9951" containerName="istio-proxy"
Oct 31 15:47:46.778964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.778944    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.832818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.832793    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:47:46.870227 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.870190    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpzp\" (UniqueName: \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.870394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.870323    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.878472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.878435    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b21d8a\""
Oct 31 15:47:46.878667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.878547    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b21d8a-dockercfg-fwntm\""
Oct 31 15:47:46.949334 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.949299    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:47:46.960921 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.960895    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:46.971638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.971608    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpzp\" (UniqueName: \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.971787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.971668    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp2l6\" (UniqueName: \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:46.971787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.971710    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:46.971930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.971830    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.972829 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.972805    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:46.996065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:46.996029    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:47:47.031376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.031306    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpzp\" (UniqueName: \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\") pod \"payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:47.072899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.072868    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp2l6\" (UniqueName: \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:47.073093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.072912    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:47.073845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.073822    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:47.093517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.093489    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:47.130012 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.129982    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp2l6\" (UniqueName: \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\") pod \"payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:47.276758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.276727    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:47.476347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.476325    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:47:47.477449 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:47:47.477422    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc3b724_08a3_473a_bf4b_466ea7cba4f6.slice/crio-b4c5fea4b9c32901dbd9f67d1e3fd49913a940714c98a9b06a1bab5e52250cba WatchSource:0}: Error finding container b4c5fea4b9c32901dbd9f67d1e3fd49913a940714c98a9b06a1bab5e52250cba: Status 404 returned error can't find the container with id b4c5fea4b9c32901dbd9f67d1e3fd49913a940714c98a9b06a1bab5e52250cba
Oct 31 15:47:47.621174 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.621136    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" event={"ID":"ebc3b724-08a3-473a-bf4b-466ea7cba4f6","Type":"ContainerStarted","Data":"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"}
Oct 31 15:47:47.621369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.621202    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" event={"ID":"ebc3b724-08a3-473a-bf4b-466ea7cba4f6","Type":"ContainerStarted","Data":"b4c5fea4b9c32901dbd9f67d1e3fd49913a940714c98a9b06a1bab5e52250cba"}
Oct 31 15:47:47.621369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.621257    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:47.656694 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.656670    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:47:47.657299 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:47:47.657272    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1627dacc_6a95_4945_944e_30fc732e9a1f.slice/crio-89919a0fccffb386417e13affbb665c66c985c65dde67bd72c26b54e0a8bbe11 WatchSource:0}: Error finding container 89919a0fccffb386417e13affbb665c66c985c65dde67bd72c26b54e0a8bbe11: Status 404 returned error can't find the container with id 89919a0fccffb386417e13affbb665c66c985c65dde67bd72c26b54e0a8bbe11
Oct 31 15:47:47.734868 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:47.734812    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" podStartSLOduration=1.7347923779999999 podStartE2EDuration="1.734792378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:47:47.733061182 +0000 UTC m=+3708.459342160" watchObservedRunningTime="2026-10-31 15:47:47.734792378 +0000 UTC m=+3708.461073357"
Oct 31 15:47:48.630888 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:48.630855    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" event={"ID":"1627dacc-6a95-4945-944e-30fc732e9a1f","Type":"ContainerStarted","Data":"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"}
Oct 31 15:47:48.631342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:48.630910    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" event={"ID":"1627dacc-6a95-4945-944e-30fc732e9a1f","Type":"ContainerStarted","Data":"89919a0fccffb386417e13affbb665c66c985c65dde67bd72c26b54e0a8bbe11"}
Oct 31 15:47:48.631342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:48.631093    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:48.745044 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:48.744988    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" podStartSLOduration=2.7449739380000002 podStartE2EDuration="2.744973938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:47:48.742612717 +0000 UTC m=+3709.468893689" watchObservedRunningTime="2026-10-31 15:47:48.744973938 +0000 UTC m=+3709.471254941"
Oct 31 15:47:54.298029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.297999    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:47:54.302246 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.302218    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.311892 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.311476    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:47:54.354154 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.354125    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-73b8d8-serving-cert\""
Oct 31 15:47:54.368385 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.368339    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6jq\" (UniqueName: \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.368557 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.368532    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.368702 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.368565    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.368775 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.368717    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.469642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.469601    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.469642 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.469646    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.469915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.469690    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.469915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.469856    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6jq\" (UniqueName: \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.470474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.470447    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.472364 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.472340    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.472645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.472625    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.530108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.530081    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6jq\" (UniqueName: \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\") pod \"maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:54.619166 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:54.619133    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:55.024015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:55.023979    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:47:55.024515 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:47:55.024485    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccba9db_066d_46bd_adb3_9a841dc5fb78.slice/crio-83df07e1e9a60e04d871d54df71055ec65a47e187cb6db317fe6a2ff4c65c112 WatchSource:0}: Error finding container 83df07e1e9a60e04d871d54df71055ec65a47e187cb6db317fe6a2ff4c65c112: Status 404 returned error can't find the container with id 83df07e1e9a60e04d871d54df71055ec65a47e187cb6db317fe6a2ff4c65c112
Oct 31 15:47:55.682914 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:55.682882    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" event={"ID":"9ccba9db-066d-46bd-adb3-9a841dc5fb78","Type":"ContainerStarted","Data":"83df07e1e9a60e04d871d54df71055ec65a47e187cb6db317fe6a2ff4c65c112"}
Oct 31 15:47:56.692264 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:56.692229    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" event={"ID":"9ccba9db-066d-46bd-adb3-9a841dc5fb78","Type":"ContainerStarted","Data":"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"}
Oct 31 15:47:56.692713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:56.692369    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:47:56.805668 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:56.805334    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" podStartSLOduration=1.935831082 podStartE2EDuration="2.805315547s" totalImagesPullingTime="869.484465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:47:56.804879704 +0000 UTC m=+3717.531160682" watchObservedRunningTime="2026-10-31 15:47:56.805315547 +0000 UTC m=+3717.531596524"
Oct 31 15:47:58.634170 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:58.634145    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:47:59.640747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.640712    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:47:59.809742 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.809711    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:47:59.813976 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.813954    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.868545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.868512    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c4840c-openshift-default-dockercfg-6x7j7\""
Oct 31 15:47:59.879161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.875759    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:47:59.940551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940454    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.940551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940526    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.940854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940610    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.940854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940728    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.940854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940766    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5nrc\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.941031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940921    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.941031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.940973    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.941031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.941005    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:47:59.941200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:47:59.941041    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047207    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047284    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047327    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047355    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5nrc\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047473    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047515    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047544    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047607    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.047694    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.048615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.048443    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.049333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.048723    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.049333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.048971    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.068601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.059743    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.068601 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.060001    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.082602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.080242    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.089735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.086262    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.104153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.104065    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.104153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.104752    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5nrc\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\") pod \"e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") " pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.130808 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.130776    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:00.571393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.571362    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:48:00.572262 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:48:00.572231    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e8519b_7412_4e11_9e40_1da6e7d43296.slice/crio-248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f WatchSource:0}: Error finding container 248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f: Status 404 returned error can't find the container with id 248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f
Oct 31 15:48:00.575262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.575227    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:00.575392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.575307    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:00.575392 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.575352    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:00.725616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.725234    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerStarted","Data":"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"}
Oct 31 15:48:00.725616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.725316    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerStarted","Data":"248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f"}
Oct 31 15:48:00.848456 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:00.848332    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podStartSLOduration=1.848317819 podStartE2EDuration="1.848317819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:00.843459309 +0000 UTC m=+3721.569740289" watchObservedRunningTime="2026-10-31 15:48:00.848317819 +0000 UTC m=+3721.574598868"
Oct 31 15:48:01.131188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:01.131098    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:01.137762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:01.137734    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:01.738983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:01.737205    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:01.738983 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:01.738484    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:48:02.706289 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:02.706254    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:48:07.012660 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:48:07.012620    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:48:12.307970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.300903    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:48:12.307970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.305270    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.351747 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.351713    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:48:12.391002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.390962    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4c9a78\""
Oct 31 15:48:12.391002 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.390972    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4c9a78-dockercfg-5rjk2\""
Oct 31 15:48:12.410762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.410729    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrwt\" (UniqueName: \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.410924 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.410793    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.512118 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.512079    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrwt\" (UniqueName: \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.512299 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.512171    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.512891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.512868    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.569727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.569653    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrwt\" (UniqueName: \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\") pod \"payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:12.623731 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:12.623692    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:13.003850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:13.003818    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:48:13.003990 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:48:13.003908    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbe4a36_e1be_4db2_9479_ae15d162e805.slice/crio-80a5e8273faac1af6afff9819755070fec1650a77243d89f87e4d012d0571159 WatchSource:0}: Error finding container 80a5e8273faac1af6afff9819755070fec1650a77243d89f87e4d012d0571159: Status 404 returned error can't find the container with id 80a5e8273faac1af6afff9819755070fec1650a77243d89f87e4d012d0571159
Oct 31 15:48:13.826085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:13.826039    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" event={"ID":"dcbe4a36-e1be-4db2-9479-ae15d162e805","Type":"ContainerStarted","Data":"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"}
Oct 31 15:48:13.826502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:13.826117    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:13.826502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:13.826139    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" event={"ID":"dcbe4a36-e1be-4db2-9479-ae15d162e805","Type":"ContainerStarted","Data":"80a5e8273faac1af6afff9819755070fec1650a77243d89f87e4d012d0571159"}
Oct 31 15:48:13.944458 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:13.944401    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" podStartSLOduration=1.944386795 podStartE2EDuration="1.944386795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:13.942859426 +0000 UTC m=+3734.669140406" watchObservedRunningTime="2026-10-31 15:48:13.944386795 +0000 UTC m=+3734.670667812"
Oct 31 15:48:24.837608 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:24.837560    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:48:37.013149 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:48:37.013109    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:48:45.303593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.303528    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:48:45.308932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.308906    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.362794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.362738    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cf95c9-openshift-default-dockercfg-qfxcp\""
Oct 31 15:48:45.450631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.450600    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:48:45.453778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.452517    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ns2p\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.454078 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.454058    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.454262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.454245    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.454368 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.454357    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.454681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.454658    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.456233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.456212    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.456422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.456405    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.456588 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.456550    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.456734 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.456719    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.558065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.557985    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.558387 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.558368    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.558549 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.558496    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.558753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.558727    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.558933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.558915    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559067    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559230    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ns2p\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559443    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559641 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.558863    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559071    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559857 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559841    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.559999 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.559986    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.560895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.560146    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.561006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.560360    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.561970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.561950    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.563411 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.563391    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.617331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.617293    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.617536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.617511    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ns2p\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\") pod \"e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") " pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:45.630182 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:45.630149    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:46.074981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:46.074941    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:48:46.115946 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:48:46.114983    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdaaa87_ee5e_4ce2_98ed_a91c7b22d4a5.slice/crio-823004ab86d2efc5ac14207b32962b911c97c4bdb082383a405089e03e0f99c7 WatchSource:0}: Error finding container 823004ab86d2efc5ac14207b32962b911c97c4bdb082383a405089e03e0f99c7: Status 404 returned error can't find the container with id 823004ab86d2efc5ac14207b32962b911c97c4bdb082383a405089e03e0f99c7
Oct 31 15:48:46.118892 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:46.118846    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:46.119029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:46.118948    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:46.119029 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:46.118987    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:48:47.062656 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:47.062602    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerStarted","Data":"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"}
Oct 31 15:48:47.063162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:47.062679    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerStarted","Data":"823004ab86d2efc5ac14207b32962b911c97c4bdb082383a405089e03e0f99c7"}
Oct 31 15:48:47.276375 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:47.276314    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podStartSLOduration=2.276299441 podStartE2EDuration="2.276299441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:47.226854247 +0000 UTC m=+3767.953135226" watchObservedRunningTime="2026-10-31 15:48:47.276299441 +0000 UTC m=+3768.002580419"
Oct 31 15:48:47.630683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:47.630640    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:47.637190 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:47.637163    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:48.071371 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:48.071339    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:48:48.072302 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:48:48.072283    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:49:07.015505 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:07.015463    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:49:31.329596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.329282    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:49:31.334884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.334857    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.354496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.354465    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.354674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.354508    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.354746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.354683    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.354746 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.354711    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtcc\" (UniqueName: \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.384048 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.384020    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:49:31.431224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.431194    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c4840c-serving-cert\""
Oct 31 15:49:31.456389 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.456354    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.456598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.456428    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.456598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.456504    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.456598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.456530    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtcc\" (UniqueName: \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.457166 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:31.457133    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-serving-cert: secret "maas-api-e2e-inf-a-c4840c-serving-cert" not found
Oct 31 15:49:31.457266 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.457172    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.457337 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:31.457274    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls podName:4c3391d6-86f4-4e10-8f48-03a8dc186414 nodeName:}" failed. No retries permitted until 2026-10-31 15:49:31.957253807 +0000 UTC m=+3812.683534767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls") pod "maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" (UID: "4c3391d6-86f4-4e10-8f48-03a8dc186414") : secret "maas-api-e2e-inf-a-c4840c-serving-cert" not found
Oct 31 15:49:31.466520 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.466427    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.533682 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.533646    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtcc\" (UniqueName: \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.965318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.965280    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:31.969824 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:31.969799    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") pod \"maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:32.252730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:32.252632    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:32.671239 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:32.671208    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:49:32.671658 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:49:32.671440    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3391d6_86f4_4e10_8f48_03a8dc186414.slice/crio-f59e993b91836b3ee1f1efdb375f3b3ec80db9868d2ca76dfcbca4ef353917c1 WatchSource:0}: Error finding container f59e993b91836b3ee1f1efdb375f3b3ec80db9868d2ca76dfcbca4ef353917c1: Status 404 returned error can't find the container with id f59e993b91836b3ee1f1efdb375f3b3ec80db9868d2ca76dfcbca4ef353917c1
Oct 31 15:49:33.452839 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:33.452804    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" event={"ID":"4c3391d6-86f4-4e10-8f48-03a8dc186414","Type":"ContainerStarted","Data":"f59e993b91836b3ee1f1efdb375f3b3ec80db9868d2ca76dfcbca4ef353917c1"}
Oct 31 15:49:34.476596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:34.475836    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" event={"ID":"4c3391d6-86f4-4e10-8f48-03a8dc186414","Type":"ContainerStarted","Data":"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"}
Oct 31 15:49:34.477683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:34.477612    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:34.596060 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:34.596013    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" podStartSLOduration=2.703193463 podStartE2EDuration="3.595997194s" totalImagesPullingTime="892.803731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:34.59295049 +0000 UTC m=+3815.319231470" watchObservedRunningTime="2026-10-31 15:49:34.595997194 +0000 UTC m=+3815.322278171"
Oct 31 15:49:36.347215 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.347172    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:49:36.351554 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.351534    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.408717 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.408683    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:49:36.440408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.440358    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.440666 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.440645    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.440741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.440730    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwvw\" (UniqueName: \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.440845 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.440828    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.452799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.452767    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4c9a78-serving-cert\""
Oct 31 15:49:36.542472 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.542429    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.542706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.542523    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwvw\" (UniqueName: \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.542706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.542587    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.542706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.542641    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.544719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.544694    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.545477 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:36.545456    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-serving-cert: secret "maas-api-e2e-inf-b-4c9a78-serving-cert" not found
Oct 31 15:49:36.545594 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:36.545521    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls podName:3cfe341c-8db2-4a11-8d3f-42e553dda630 nodeName:}" failed. No retries permitted until 2026-10-31 15:49:37.045502296 +0000 UTC m=+3817.771783258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls") pod "maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" (UID: "3cfe341c-8db2-4a11-8d3f-42e553dda630") : secret "maas-api-e2e-inf-b-4c9a78-serving-cert" not found
Oct 31 15:49:36.549123 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.549101    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:36.602163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:36.602094    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwvw\" (UniqueName: \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:37.013349 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:49:37.013257    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:49:37.050800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:37.050755    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:37.053362 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:37.053337    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") pod \"maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:37.268176 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:37.268135    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:37.656408 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:37.656377    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:49:37.657097 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:49:37.657068    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfe341c_8db2_4a11_8d3f_42e553dda630.slice/crio-26a7a618511385031f6b68573a80e69dde639979cd962922f5999f16877857ce WatchSource:0}: Error finding container 26a7a618511385031f6b68573a80e69dde639979cd962922f5999f16877857ce: Status 404 returned error can't find the container with id 26a7a618511385031f6b68573a80e69dde639979cd962922f5999f16877857ce
Oct 31 15:49:38.512645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.512590    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" event={"ID":"3cfe341c-8db2-4a11-8d3f-42e553dda630","Type":"ContainerStarted","Data":"26a7a618511385031f6b68573a80e69dde639979cd962922f5999f16877857ce"}
Oct 31 15:49:38.513647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.513522    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac093c8c-7308-4130-88dc-5eba33da8976" containerID="6f434f65b55fe704969d3e8340c895bc70e59dc080a476dfab0be5cf8a19d476" exitCode=0
Oct 31 15:49:38.513647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.513554    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerDied","Data":"6f434f65b55fe704969d3e8340c895bc70e59dc080a476dfab0be5cf8a19d476"}
Oct 31 15:49:38.514097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.514075    4549 scope.go:122] "RemoveContainer" containerID="6f434f65b55fe704969d3e8340c895bc70e59dc080a476dfab0be5cf8a19d476"
Oct 31 15:49:38.516297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.516262    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:49:38.516409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.516351    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:49:38.516409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:38.516391    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:49:39.543300 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.543262    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerStarted","Data":"299bf577262d1021f162c1969d9bff909edbd57207c1eb26b124668f2846d141"}
Oct 31 15:49:39.545039 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.545002    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" event={"ID":"3cfe341c-8db2-4a11-8d3f-42e553dda630","Type":"ContainerStarted","Data":"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"}
Oct 31 15:49:39.545165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.545072    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:49:39.777097 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.777032    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" podStartSLOduration=3.011773581 podStartE2EDuration="3.777012599s" totalImagesPullingTime="765.239018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:39.774939528 +0000 UTC m=+3820.501220538" watchObservedRunningTime="2026-10-31 15:49:39.777012599 +0000 UTC m=+3820.503293614"
Oct 31 15:49:39.875563 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.875477    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:49:39.876386 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.876355    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 15:49:39.876519 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:39.876431    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 15:49:40.875813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:40.875726    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 15:49:40.875813 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:40.875796    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 15:49:41.496050 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:41.496021    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:49:41.574719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:41.574687    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:49:41.875981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:41.875936    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 15:49:41.876412 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:41.876004    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 15:49:42.875347 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:42.875309    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 15:49:42.875527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:42.875358    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 15:49:43.875596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:43.875541    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Oct 31 15:49:43.876006 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:43.875637    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Oct 31 15:49:44.898866 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:44.898835    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:49:44.900993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:44.900967    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:49:45.558009 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:49:45.557982    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:50:01.738507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.738469    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:01.739065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.738539    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:01.828287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.828244    4549 generic.go:356] "Generic (PLEG): container finished" podID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerID="9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b" exitCode=0
Oct 31 15:50:01.828466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.828292    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerDied","Data":"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"}
Oct 31 15:50:01.828875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.828854    4549 scope.go:122] "RemoveContainer" containerID="9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"
Oct 31 15:50:01.830800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.830770    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:01.830882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.830848    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:01.830926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:01.830901    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:02.840125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:02.840086    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerStarted","Data":"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"}
Oct 31 15:50:03.131589 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:03.131481    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:50:03.131782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:03.131658    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:03.131782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:03.131700    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:04.132153 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:04.132113    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:04.132651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:04.132186    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:05.131252 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:05.131216    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:05.131473 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:05.131280    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:06.131350 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:06.131313    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:06.131799 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:06.131373    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:07.032976 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:50:07.032938    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:50:07.131639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:07.131599    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:07.132066 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:07.131665    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:08.131653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:08.131614    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 31 15:50:08.132260 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:08.131684    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 31 15:50:09.136514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:09.136486    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:50:09.136981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:09.136961    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:50:09.137763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:09.137742    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:50:37.012166 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:50:37.012128    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:50:44.310199 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:44.310163    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:50:44.310773 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:44.310367    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy" containerID="cri-o://3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad" gracePeriod=30
Oct 31 15:50:46.665349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:46.665311    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:50:46.773143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:46.773108    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:50:49.573363 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.573339    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:50:49.671660 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.671540    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.671933 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.671911    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672067 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672052    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672184 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672171    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672300    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672459 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672445    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672613 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672563    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.672898 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.672881    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.673051 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.673038    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\") pod \"56763e72-bc6c-491e-abbd-52dfce2d13dc\" (UID: \"56763e72-bc6c-491e-abbd-52dfce2d13dc\") "
Oct 31 15:50:49.674027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.674003    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:50:49.674355 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.674338    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:50:49.676500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.675845    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:50:49.676716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.676066    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:50:49.676816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.676221    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:50:49.680962 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.680928    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:50:49.681526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.681506    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:50:49.690802 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.690735    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:50:49.692777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.692717    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6" pod "56763e72-bc6c-491e-abbd-52dfce2d13dc" (UID: "56763e72-bc6c-491e-abbd-52dfce2d13dc"). InnerVolumeSpecName "kube-api-access-xpmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794593    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794629    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794645    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794659    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794673    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56763e72-bc6c-491e-abbd-52dfce2d13dc-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794685    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794699    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpmr6\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-kube-api-access-xpmr6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794715    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:49.794755 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:49.794730    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56763e72-bc6c-491e-abbd-52dfce2d13dc-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:50.249158 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.249120    4549 generic.go:356] "Generic (PLEG): container finished" podID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerID="3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad" exitCode=0
Oct 31 15:50:50.249372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.249177    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"
Oct 31 15:50:50.249372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.249178    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" event={"ID":"56763e72-bc6c-491e-abbd-52dfce2d13dc","Type":"ContainerDied","Data":"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"}
Oct 31 15:50:50.249372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.249289    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq" event={"ID":"56763e72-bc6c-491e-abbd-52dfce2d13dc","Type":"ContainerDied","Data":"2c54f92dcc4ebc4a02136ebf08dd6cabe1faa3fdd6957edd594664c0140ec489"}
Oct 31 15:50:50.249372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.249309    4549 scope.go:122] "RemoveContainer" containerID="3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"
Oct 31 15:50:50.261121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.261100    4549 scope.go:122] "RemoveContainer" containerID="3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"
Oct 31 15:50:50.261395 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:50:50.261372    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad\": container with ID starting with 3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad not found: ID does not exist" containerID="3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"
Oct 31 15:50:50.261466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.261402    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad"} err="failed to get container status \"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad\": rpc error: code = NotFound desc = could not find container \"3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad\": container with ID starting with 3c8455827de87aa948bf02c34495ad781acd23c214cfc42bebb05148ee779cad not found: ID does not exist"
Oct 31 15:50:50.488630 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.488593    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:50:50.538494 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:50.538452    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-s7xxq"]
Oct 31 15:50:51.430638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:51.430603    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" path="/var/lib/kubelet/pods/56763e72-bc6c-491e-abbd-52dfce2d13dc/volumes"
Oct 31 15:50:51.666259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:51.666168    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" podUID="1627dacc-6a95-4945-944e-30fc732e9a1f" containerName="payload-processing" containerID="cri-o://b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6" gracePeriod=25
Oct 31 15:50:51.774402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:51.774363    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" podUID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" containerName="payload-pre-processing" containerID="cri-o://5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44" gracePeriod=25
Oct 31 15:50:52.032035 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.031681    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:50:52.154678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.154651    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\") pod \"1627dacc-6a95-4945-944e-30fc732e9a1f\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") "
Oct 31 15:50:52.155069 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.155040    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume" pod "1627dacc-6a95-4945-944e-30fc732e9a1f" (UID: "1627dacc-6a95-4945-944e-30fc732e9a1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:50:52.155189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.155158    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\" (UniqueName: \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\") pod \"1627dacc-6a95-4945-944e-30fc732e9a1f\" (UID: \"1627dacc-6a95-4945-944e-30fc732e9a1f\") "
Oct 31 15:50:52.155711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.155689    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1627dacc-6a95-4945-944e-30fc732e9a1f-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:52.157935 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.157912    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6" pod "1627dacc-6a95-4945-944e-30fc732e9a1f" (UID: "1627dacc-6a95-4945-944e-30fc732e9a1f"). InnerVolumeSpecName "kube-api-access-hp2l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:50:52.159997 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.159970    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:50:52.256429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.256335    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\" (UniqueName: \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\") pod \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") "
Oct 31 15:50:52.256429 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.256380    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\") pod \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\" (UID: \"ebc3b724-08a3-473a-bf4b-466ea7cba4f6\") "
Oct 31 15:50:52.256828 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.256802    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume" pod "ebc3b724-08a3-473a-bf4b-466ea7cba4f6" (UID: "ebc3b724-08a3-473a-bf4b-466ea7cba4f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:50:52.257008 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.256988    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp2l6\" (UniqueName: \"kubernetes.io/projected/1627dacc-6a95-4945-944e-30fc732e9a1f-kube-api-access-hp2l6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:52.257008 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.257007    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:52.258765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.258741    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp" pod "ebc3b724-08a3-473a-bf4b-466ea7cba4f6" (UID: "ebc3b724-08a3-473a-bf4b-466ea7cba4f6"). InnerVolumeSpecName "kube-api-access-bwpzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:50:52.264713 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.264689    4549 generic.go:356] "Generic (PLEG): container finished" podID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" containerID="5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44" exitCode=0
Oct 31 15:50:52.264814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.264739    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"
Oct 31 15:50:52.264814 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.264773    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" event={"ID":"ebc3b724-08a3-473a-bf4b-466ea7cba4f6","Type":"ContainerDied","Data":"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"}
Oct 31 15:50:52.264938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.264830    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn" event={"ID":"ebc3b724-08a3-473a-bf4b-466ea7cba4f6","Type":"ContainerDied","Data":"b4c5fea4b9c32901dbd9f67d1e3fd49913a940714c98a9b06a1bab5e52250cba"}
Oct 31 15:50:52.264938 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.264857    4549 scope.go:122] "RemoveContainer" containerID="5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"
Oct 31 15:50:52.266365 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.266341    4549 generic.go:356] "Generic (PLEG): container finished" podID="1627dacc-6a95-4945-944e-30fc732e9a1f" containerID="b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6" exitCode=0
Oct 31 15:50:52.266466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.266439    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" event={"ID":"1627dacc-6a95-4945-944e-30fc732e9a1f","Type":"ContainerDied","Data":"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"}
Oct 31 15:50:52.266466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.266445    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"
Oct 31 15:50:52.266611 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.266481    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx" event={"ID":"1627dacc-6a95-4945-944e-30fc732e9a1f","Type":"ContainerDied","Data":"89919a0fccffb386417e13affbb665c66c985c65dde67bd72c26b54e0a8bbe11"}
Oct 31 15:50:52.276216 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.276195    4549 scope.go:122] "RemoveContainer" containerID="5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"
Oct 31 15:50:52.276515 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:50:52.276496    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44\": container with ID starting with 5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44 not found: ID does not exist" containerID="5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"
Oct 31 15:50:52.276602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.276523    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44"} err="failed to get container status \"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44\": rpc error: code = NotFound desc = could not find container \"5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44\": container with ID starting with 5a85cdb81636ee297cbec08a3096c156eb001b1fdbe74cbe0600149224c63d44 not found: ID does not exist"
Oct 31 15:50:52.276602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.276539    4549 scope.go:122] "RemoveContainer" containerID="b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"
Oct 31 15:50:52.339593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.337072    4549 scope.go:122] "RemoveContainer" containerID="b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"
Oct 31 15:50:52.350592 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:50:52.345015    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6\": container with ID starting with b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6 not found: ID does not exist" containerID="b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"
Oct 31 15:50:52.350592 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.345062    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6"} err="failed to get container status \"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6\": rpc error: code = NotFound desc = could not find container \"b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6\": container with ID starting with b3cede1f4d981fffaeaa4274f0f5a203fc8b35337d8a2a824321bd6ee20d6fb6 not found: ID does not exist"
Oct 31 15:50:52.358487 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.358457    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpzp\" (UniqueName: \"kubernetes.io/projected/ebc3b724-08a3-473a-bf4b-466ea7cba4f6-kube-api-access-bwpzp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:50:52.405596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.403312    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:50:52.414598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.414123    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b21d8a-758b46c989-rqhkx"]
Oct 31 15:50:52.531022 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.530991    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:50:52.535098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:52.535073    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b21d8a-855f4bc467-lwjqn"]
Oct 31 15:50:53.433496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:53.433454    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1627dacc-6a95-4945-944e-30fc732e9a1f" path="/var/lib/kubelet/pods/1627dacc-6a95-4945-944e-30fc732e9a1f/volumes"
Oct 31 15:50:53.434033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:53.434011    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" path="/var/lib/kubelet/pods/ebc3b724-08a3-473a-bf4b-466ea7cba4f6/volumes"
Oct 31 15:50:55.009604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:55.009156    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:50:55.009604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:55.009410    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" containerID="cri-o://299bf577262d1021f162c1969d9bff909edbd57207c1eb26b124668f2846d141" gracePeriod=30
Oct 31 15:50:59.347010 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.346978    4549 generic.go:356] "Generic (PLEG): container finished" podID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerID="240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762" exitCode=0
Oct 31 15:50:59.347484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.347026    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerDied","Data":"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"}
Oct 31 15:50:59.347484 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.347424    4549 scope.go:122] "RemoveContainer" containerID="240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"
Oct 31 15:50:59.349283 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.349250    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:59.349390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.349333    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:59.349390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.349368    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:50:59.901638 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.901605    4549 patch_prober.go:29] interesting pod/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:50:59.901816 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:50:59.901662    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:51:00.356303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.356275    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac093c8c-7308-4130-88dc-5eba33da8976" containerID="299bf577262d1021f162c1969d9bff909edbd57207c1eb26b124668f2846d141" exitCode=0
Oct 31 15:51:00.356722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.356349    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerDied","Data":"299bf577262d1021f162c1969d9bff909edbd57207c1eb26b124668f2846d141"}
Oct 31 15:51:00.356722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.356411    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4" event={"ID":"ac093c8c-7308-4130-88dc-5eba33da8976","Type":"ContainerDied","Data":"a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a"}
Oct 31 15:51:00.356722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.356443    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a92618d0a4e2809cf31212b9acb6e64d79990fe1807587aad1b97417d8d27c7a"
Oct 31 15:51:00.356722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.356462    4549 scope.go:122] "RemoveContainer" containerID="6f434f65b55fe704969d3e8340c895bc70e59dc080a476dfab0be5cf8a19d476"
Oct 31 15:51:00.358804 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.358784    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerStarted","Data":"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"}
Oct 31 15:51:00.371795 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.371777    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:51:00.472968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.472938    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473096    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473124    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473168 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473159    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473188    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473220    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473227    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473266    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473327    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:00.473353 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473347    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473415    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\") pod \"ac093c8c-7308-4130-88dc-5eba33da8976\" (UID: \"ac093c8c-7308-4130-88dc-5eba33da8976\") "
Oct 31 15:51:00.473729 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473459    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:00.474141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.473756    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:00.474422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474401    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:51:00.475453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474643    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.475453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474669    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.475453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474685    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.475453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474700    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac093c8c-7308-4130-88dc-5eba33da8976-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.475453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.474716    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.476819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.476791    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:51:00.477321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.477287    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "kube-api-access-jv95l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:51:00.477615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.477595    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:51:00.477852 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.477831    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy" pod "ac093c8c-7308-4130-88dc-5eba33da8976" (UID: "ac093c8c-7308-4130-88dc-5eba33da8976"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:00.575647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.575605    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac093c8c-7308-4130-88dc-5eba33da8976-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.575647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.575644    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac093c8c-7308-4130-88dc-5eba33da8976-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.575647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.575655    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.575903 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.575664    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv95l\" (UniqueName: \"kubernetes.io/projected/ac093c8c-7308-4130-88dc-5eba33da8976-kube-api-access-jv95l\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:00.636786 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.636749    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:51:00.637163 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.637142    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:00.637313 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:00.637290    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:01.367505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:01.367473    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"
Oct 31 15:51:01.631162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:01.631058    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:01.631162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:01.631120    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:01.653004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:01.652970    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:51:01.658464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:01.658426    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b21d8a-openshift-default-6f8dc878ff-lr8w4"]
Oct 31 15:51:02.631234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:02.631192    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:02.631798 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:02.631269    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:03.071258 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.071224    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:51:03.081258 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.081237    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:51:03.259437 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.259413    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:51:03.436683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.436616    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" path="/var/lib/kubelet/pods/ac093c8c-7308-4130-88dc-5eba33da8976/volumes"
Oct 31 15:51:03.632612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.632063    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:03.632612 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:03.632123    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:04.630979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:04.630942    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:04.631165 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:04.631011    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:05.631094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:05.631056    4549 patch_prober.go:29] interesting pod/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 31 15:51:05.631527 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:05.631123    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 31 15:51:06.635838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:06.635805    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:51:06.636777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:06.636758    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:51:07.012661 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:51:07.012552    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:51:26.097327 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:26.097252    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:51:26.097907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:26.097459    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy" containerID="cri-o://c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86" gracePeriod=30
Oct 31 15:51:31.472457 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.472426    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:51:31.584394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.584357    4549 generic.go:356] "Generic (PLEG): container finished" podID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerID="c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86" exitCode=0
Oct 31 15:51:31.584616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.584405    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"
Oct 31 15:51:31.584616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.584409    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" event={"ID":"a1c708df-ee72-4192-8b59-0fe3ba98b852","Type":"ContainerDied","Data":"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"}
Oct 31 15:51:31.584616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.584451    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp" event={"ID":"a1c708df-ee72-4192-8b59-0fe3ba98b852","Type":"ContainerDied","Data":"34e4f160b1722b1d974a390e7174ab53f5d423658dcd29537894396944587256"}
Oct 31 15:51:31.584616 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.584472    4549 scope.go:122] "RemoveContainer" containerID="c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"
Oct 31 15:51:31.595319 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.595296    4549 scope.go:122] "RemoveContainer" containerID="c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"
Oct 31 15:51:31.595616 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:51:31.595586    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86\": container with ID starting with c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86 not found: ID does not exist" containerID="c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"
Oct 31 15:51:31.595698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.595631    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86"} err="failed to get container status \"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86\": rpc error: code = NotFound desc = could not find container \"c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86\": container with ID starting with c9c191fdc647566fb4a4532f9d24fe191c21eeb97247dfce6a9fe4d7e315cf86 not found: ID does not exist"
Oct 31 15:51:31.624604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624539    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.624833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624672    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.624833 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624792    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.624987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624851    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.624987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624895    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.624987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.624981    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.625162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.625037    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.625162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.625091    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.625162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.625149    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\") pod \"a1c708df-ee72-4192-8b59-0fe3ba98b852\" (UID: \"a1c708df-ee72-4192-8b59-0fe3ba98b852\") "
Oct 31 15:51:31.627850 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.627551    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:51:31.628331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.628118    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:31.628462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.628332    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:31.628703 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.628680    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:31.628908 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.628889    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:31.657004 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.656674    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:51:31.665251 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.665193    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "kube-api-access-vm5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:51:31.665614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.665559    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:51:31.668496 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.668447    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy" pod "a1c708df-ee72-4192-8b59-0fe3ba98b852" (UID: "a1c708df-ee72-4192-8b59-0fe3ba98b852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727335    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727363    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm5xn\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-kube-api-access-vm5xn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727373    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c708df-ee72-4192-8b59-0fe3ba98b852-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727388    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727401    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727409    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727417    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727425    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c708df-ee72-4192-8b59-0fe3ba98b852-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:31.727454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:31.727434    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c708df-ee72-4192-8b59-0fe3ba98b852-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:51:32.010125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:32.010026    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:51:32.060683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:32.060632    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-bnhfp"]
Oct 31 15:51:33.431018 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:33.430981    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" path="/var/lib/kubelet/pods/a1c708df-ee72-4192-8b59-0fe3ba98b852/volumes"
Oct 31 15:51:37.012338 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:51:37.012300    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:51:44.786657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.786619    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 15:51:44.787200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787185    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787200    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787220    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1627dacc-6a95-4945-944e-30fc732e9a1f" containerName="payload-processing"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787225    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1627dacc-6a95-4945-944e-30fc732e9a1f" containerName="payload-processing"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787233    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787238    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787249    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787254    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787268    4549 cpu_manager.go:414] "containerMap: removing container" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787272    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787281    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" containerName="payload-pre-processing"
Oct 31 15:51:44.787312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787285    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" containerName="payload-pre-processing"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787368    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787378    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1627dacc-6a95-4945-944e-30fc732e9a1f" containerName="payload-processing"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787386    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac093c8c-7308-4130-88dc-5eba33da8976" containerName="istio-proxy"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787392    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c708df-ee72-4192-8b59-0fe3ba98b852" containerName="istio-proxy"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787399    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="56763e72-bc6c-491e-abbd-52dfce2d13dc" containerName="istio-proxy"
Oct 31 15:51:44.787657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.787405    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc3b724-08a3-473a-bf4b-466ea7cba4f6" containerName="payload-pre-processing"
Oct 31 15:51:44.790879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.790859    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.892926 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.892890    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.892946    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893024    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893101 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893079    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893196    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893243    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8vk\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893373 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893283    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893426    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.893505 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.893490    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.993763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.993725    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 15:51:44.994316 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994285    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994342    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994394    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994419    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994548    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994592    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994616    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8vk\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994646    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.994937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994800    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.995186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.995160    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.995328 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.995282    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.995544 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.994901    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.995544 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.995522    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.995745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.995684    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.997428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.997408    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:44.997428 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:44.997423    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:45.054133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.054067    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:45.054273 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.054231    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8vk\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\") pod \"maas-default-gateway-openshift-default-67bf986f99-dnbd5\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:45.106561 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.106521    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:45.533342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.533311    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 15:51:45.534177 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:51:45.534153    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b63b5b_d342_499b_80d3_f9d241157934.slice/crio-2aaf6c2a5824f7cc67d22fa8e1c0cfa0653363a9849dcdfe0b3b47d806db2216 WatchSource:0}: Error finding container 2aaf6c2a5824f7cc67d22fa8e1c0cfa0653363a9849dcdfe0b3b47d806db2216: Status 404 returned error can't find the container with id 2aaf6c2a5824f7cc67d22fa8e1c0cfa0653363a9849dcdfe0b3b47d806db2216
Oct 31 15:51:45.536895 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.536863    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:51:45.536978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.536931    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:51:45.536978 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.536959    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:51:45.688791 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.688758    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" event={"ID":"71b63b5b-d342-499b-80d3-f9d241157934","Type":"ContainerStarted","Data":"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"}
Oct 31 15:51:45.688984 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.688820    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" event={"ID":"71b63b5b-d342-499b-80d3-f9d241157934","Type":"ContainerStarted","Data":"2aaf6c2a5824f7cc67d22fa8e1c0cfa0653363a9849dcdfe0b3b47d806db2216"}
Oct 31 15:51:45.807647 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:45.807503    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podStartSLOduration=1.80748281 podStartE2EDuration="1.80748281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:51:45.802439879 +0000 UTC m=+3946.528720870" watchObservedRunningTime="2026-10-31 15:51:45.80748281 +0000 UTC m=+3946.533763789"
Oct 31 15:51:46.107341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:46.107244    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:46.108723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:46.108692    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:46.108884 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:46.108747    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:47.110026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:47.109983    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:47.110700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:47.110661    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:48.115596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:48.107978    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:48.115596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:48.108045    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:49.118954 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:49.118914    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:49.119547 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:49.119514    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:50.119595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:50.118946    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:50.119595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:50.119009    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:51.122598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:51.120825    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:51.122598 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:51.120889    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:52.107637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:52.107562    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:52.107831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:52.107669    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:53.107357 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:53.107323    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 31 15:51:53.107875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:53.107380    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 31 15:51:55.145233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:55.145191    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:51:55.145691 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:55.145255    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:51:55.185000 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:55.184957    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:55.185760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:55.185735    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:51:55.186526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:51:55.186500    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 15:52:00.461650 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:00.461605    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:52:00.568780 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:00.568750    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:52:05.462653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.462611    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" podUID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" containerName="payload-processing" containerID="cri-o://4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0" gracePeriod=25
Oct 31 15:52:05.569885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.569811    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" podUID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" containerName="payload-pre-processing" containerID="cri-o://f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a" gracePeriod=25
Oct 31 15:52:05.733082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.733007    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:52:05.859126 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.859095    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:52:05.860143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.860101    4549 generic.go:356] "Generic (PLEG): container finished" podID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" containerID="4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0" exitCode=0
Oct 31 15:52:05.860143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.860142    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"
Oct 31 15:52:05.860360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.860174    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" event={"ID":"0d00d0ce-58fa-445e-9b1a-83ea6414eacf","Type":"ContainerDied","Data":"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"}
Oct 31 15:52:05.860360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.860247    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk" event={"ID":"0d00d0ce-58fa-445e-9b1a-83ea6414eacf","Type":"ContainerDied","Data":"1dbaed2906036dabfa6464e11f7a598923cc3d66031e7c1114de520e48ca1469"}
Oct 31 15:52:05.860360 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.860279    4549 scope.go:122] "RemoveContainer" containerID="4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"
Oct 31 15:52:05.861941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.861916    4549 generic.go:356] "Generic (PLEG): container finished" podID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" containerID="f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a" exitCode=0
Oct 31 15:52:05.862102 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.862070    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" event={"ID":"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d","Type":"ContainerDied","Data":"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"}
Oct 31 15:52:05.862180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.862107    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49" event={"ID":"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d","Type":"ContainerDied","Data":"9c1e45367571e609ccf71999a7d85c5fe3182286db3f6c8b482af819b2a92ac5"}
Oct 31 15:52:05.862180 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.862155    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"
Oct 31 15:52:05.868700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.868679    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\") pod \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") "
Oct 31 15:52:05.868818 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.868755    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\" (UniqueName: \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\") pod \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\" (UID: \"0d00d0ce-58fa-445e-9b1a-83ea6414eacf\") "
Oct 31 15:52:05.869094 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.869072    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume" pod "0d00d0ce-58fa-445e-9b1a-83ea6414eacf" (UID: "0d00d0ce-58fa-445e-9b1a-83ea6414eacf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:05.869435 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.869416    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:05.871468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.871442    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq" pod "0d00d0ce-58fa-445e-9b1a-83ea6414eacf" (UID: "0d00d0ce-58fa-445e-9b1a-83ea6414eacf"). InnerVolumeSpecName "kube-api-access-lkzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:05.872224 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.872200    4549 scope.go:122] "RemoveContainer" containerID="4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"
Oct 31 15:52:05.873622 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:05.872626    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0\": container with ID starting with 4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0 not found: ID does not exist" containerID="4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"
Oct 31 15:52:05.873622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.872662    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0"} err="failed to get container status \"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0\": rpc error: code = NotFound desc = could not find container \"4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0\": container with ID starting with 4fee54bc4738c5e3173b553c9290568f3068e9336d48577e3e33f38fecb779b0 not found: ID does not exist"
Oct 31 15:52:05.873622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.872685    4549 scope.go:122] "RemoveContainer" containerID="f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"
Oct 31 15:52:05.884441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.884387    4549 scope.go:122] "RemoveContainer" containerID="f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"
Oct 31 15:52:05.884780 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:05.884757    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a\": container with ID starting with f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a not found: ID does not exist" containerID="f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"
Oct 31 15:52:05.884872 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.884789    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a"} err="failed to get container status \"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a\": rpc error: code = NotFound desc = could not find container \"f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a\": container with ID starting with f9eb95668386034414a87ac62d4e408283fc79059bde541af8a254214af42c2a not found: ID does not exist"
Oct 31 15:52:05.971070 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.971015    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\") pod \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") "
Oct 31 15:52:05.971723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.971132    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\" (UniqueName: \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\") pod \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\" (UID: \"8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d\") "
Oct 31 15:52:05.971723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.971661    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume" pod "8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" (UID: "8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:05.971723 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.971675    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkzjq\" (UniqueName: \"kubernetes.io/projected/0d00d0ce-58fa-445e-9b1a-83ea6414eacf-kube-api-access-lkzjq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:05.974417 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:05.974372    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv" pod "8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" (UID: "8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d"). InnerVolumeSpecName "kube-api-access-9l6gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:06.072771 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.072737    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l6gv\" (UniqueName: \"kubernetes.io/projected/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-kube-api-access-9l6gv\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:06.072771 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.072773    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:06.285883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.285848    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:52:06.289003 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.288981    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-883697-794879d99f-mkr49"]
Oct 31 15:52:06.397015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.396902    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:52:06.400858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:06.400829    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-883697-7df7f5bfff-2wrtk"]
Oct 31 15:52:07.013209 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:07.013176    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:52:07.422670 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:07.422632    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" path="/var/lib/kubelet/pods/0d00d0ce-58fa-445e-9b1a-83ea6414eacf/volumes"
Oct 31 15:52:07.423232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:07.423214    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" path="/var/lib/kubelet/pods/8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d/volumes"
Oct 31 15:52:37.018493 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:37.018452    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:52:47.396119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:47.396039    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:52:47.502410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:47.502366    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:52:49.785748 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:49.785713    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:52:50.628595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:50.624174    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:52:50.628595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:50.624413    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" podUID="3cfe341c-8db2-4a11-8d3f-42e553dda630" containerName="maas-api" containerID="cri-o://9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80" gracePeriod=30
Oct 31 15:52:51.076333 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.076299    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:52:51.173146 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173111    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\") pod \"3cfe341c-8db2-4a11-8d3f-42e553dda630\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") "
Oct 31 15:52:51.173324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173166    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\") pod \"3cfe341c-8db2-4a11-8d3f-42e553dda630\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") "
Oct 31 15:52:51.173324 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173319    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\" (UniqueName: \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\") pod \"3cfe341c-8db2-4a11-8d3f-42e553dda630\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") "
Oct 31 15:52:51.173421 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173336    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") pod \"3cfe341c-8db2-4a11-8d3f-42e553dda630\" (UID: \"3cfe341c-8db2-4a11-8d3f-42e553dda630\") "
Oct 31 15:52:51.173550 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173526    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca" pod "3cfe341c-8db2-4a11-8d3f-42e553dda630" (UID: "3cfe341c-8db2-4a11-8d3f-42e553dda630"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:51.173807 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.173782    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe341c-8db2-4a11-8d3f-42e553dda630-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:51.175410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.175382    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs" pod "3cfe341c-8db2-4a11-8d3f-42e553dda630" (UID: "3cfe341c-8db2-4a11-8d3f-42e553dda630"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:52:51.175624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.175555    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls" pod "3cfe341c-8db2-4a11-8d3f-42e553dda630" (UID: "3cfe341c-8db2-4a11-8d3f-42e553dda630"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:52:51.175624 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.175598    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw" pod "3cfe341c-8db2-4a11-8d3f-42e553dda630" (UID: "3cfe341c-8db2-4a11-8d3f-42e553dda630"). InnerVolumeSpecName "kube-api-access-cwwvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:51.275815 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.275766    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwwvw\" (UniqueName: \"kubernetes.io/projected/3cfe341c-8db2-4a11-8d3f-42e553dda630-kube-api-access-cwwvw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:51.276151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.276131    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:51.276285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.276272    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cfe341c-8db2-4a11-8d3f-42e553dda630-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:51.277189 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.277161    4549 generic.go:356] "Generic (PLEG): container finished" podID="3cfe341c-8db2-4a11-8d3f-42e553dda630" containerID="9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80" exitCode=0
Oct 31 15:52:51.277617 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.277217    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"
Oct 31 15:52:51.278643 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.277239    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" event={"ID":"3cfe341c-8db2-4a11-8d3f-42e553dda630","Type":"ContainerDied","Data":"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"}
Oct 31 15:52:51.278750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.278718    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf" event={"ID":"3cfe341c-8db2-4a11-8d3f-42e553dda630","Type":"ContainerDied","Data":"26a7a618511385031f6b68573a80e69dde639979cd962922f5999f16877857ce"}
Oct 31 15:52:51.278819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.278762    4549 scope.go:122] "RemoveContainer" containerID="9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"
Oct 31 15:52:51.302267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.302239    4549 scope.go:122] "RemoveContainer" containerID="9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"
Oct 31 15:52:51.302663 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:51.302635    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80\": container with ID starting with 9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80 not found: ID does not exist" containerID="9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"
Oct 31 15:52:51.302733 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.302678    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80"} err="failed to get container status \"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80\": rpc error: code = NotFound desc = could not find container \"9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80\": container with ID starting with 9af885db7395b3775771c4fed7e4e1c8621a258be2dc11058ab569269d2a7f80 not found: ID does not exist"
Oct 31 15:52:51.452430 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.452384    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:52:51.458106 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:51.458055    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4c9a78-f997b6d94-9zlgf"]
Oct 31 15:52:52.396953 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.396908    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" podUID="719f3fc4-7f98-451d-93df-431cf106fb63" containerName="payload-processing" containerID="cri-o://f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e" gracePeriod=25
Oct 31 15:52:52.502994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.502945    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" podUID="b1911d49-ec44-451b-ad14-27ba00f49151" containerName="payload-pre-processing" containerID="cri-o://087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a" gracePeriod=25
Oct 31 15:52:52.698514 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.698483    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:52:52.790410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.790381    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:52:52.800517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.800494    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\" (UniqueName: \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\") pod \"719f3fc4-7f98-451d-93df-431cf106fb63\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") "
Oct 31 15:52:52.800686 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.800538    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\") pod \"719f3fc4-7f98-451d-93df-431cf106fb63\" (UID: \"719f3fc4-7f98-451d-93df-431cf106fb63\") "
Oct 31 15:52:52.800988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.800963    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume" pod "719f3fc4-7f98-451d-93df-431cf106fb63" (UID: "719f3fc4-7f98-451d-93df-431cf106fb63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:52.802793 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.802770    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw" pod "719f3fc4-7f98-451d-93df-431cf106fb63" (UID: "719f3fc4-7f98-451d-93df-431cf106fb63"). InnerVolumeSpecName "kube-api-access-crsjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:52.902303 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.902270    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\" (UniqueName: \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\") pod \"b1911d49-ec44-451b-ad14-27ba00f49151\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") "
Oct 31 15:52:52.902615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.902399    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\") pod \"b1911d49-ec44-451b-ad14-27ba00f49151\" (UID: \"b1911d49-ec44-451b-ad14-27ba00f49151\") "
Oct 31 15:52:52.902883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.902859    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/719f3fc4-7f98-451d-93df-431cf106fb63-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:52.902883 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.902870    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume" pod "b1911d49-ec44-451b-ad14-27ba00f49151" (UID: "b1911d49-ec44-451b-ad14-27ba00f49151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:52.903046 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.902884    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crsjw\" (UniqueName: \"kubernetes.io/projected/719f3fc4-7f98-451d-93df-431cf106fb63-kube-api-access-crsjw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:52.904788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:52.904759    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx" pod "b1911d49-ec44-451b-ad14-27ba00f49151" (UID: "b1911d49-ec44-451b-ad14-27ba00f49151"). InnerVolumeSpecName "kube-api-access-dmfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:53.004476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.004383    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmfkx\" (UniqueName: \"kubernetes.io/projected/b1911d49-ec44-451b-ad14-27ba00f49151-kube-api-access-dmfkx\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:53.004476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.004418    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1911d49-ec44-451b-ad14-27ba00f49151-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:53.293750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.293704    4549 generic.go:356] "Generic (PLEG): container finished" podID="719f3fc4-7f98-451d-93df-431cf106fb63" containerID="f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e" exitCode=0
Oct 31 15:52:53.293950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.293757    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"
Oct 31 15:52:53.293950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.293773    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" event={"ID":"719f3fc4-7f98-451d-93df-431cf106fb63","Type":"ContainerDied","Data":"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"}
Oct 31 15:52:53.293950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.293837    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k" event={"ID":"719f3fc4-7f98-451d-93df-431cf106fb63","Type":"ContainerDied","Data":"f4a5f3425cff9546409cbe2c3b3b9a2b62942f72a616702d253f861d4164c801"}
Oct 31 15:52:53.293950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.293871    4549 scope.go:122] "RemoveContainer" containerID="f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"
Oct 31 15:52:53.295465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.295442    4549 generic.go:356] "Generic (PLEG): container finished" podID="b1911d49-ec44-451b-ad14-27ba00f49151" containerID="087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a" exitCode=0
Oct 31 15:52:53.295657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.295485    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"
Oct 31 15:52:53.295657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.295614    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" event={"ID":"b1911d49-ec44-451b-ad14-27ba00f49151","Type":"ContainerDied","Data":"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"}
Oct 31 15:52:53.295657 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.295651    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk" event={"ID":"b1911d49-ec44-451b-ad14-27ba00f49151","Type":"ContainerDied","Data":"c17efc2de276a92efdfd1ea8fd251861fb24d32442457eadad79a2e225e56112"}
Oct 31 15:52:53.309098 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.309074    4549 scope.go:122] "RemoveContainer" containerID="f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"
Oct 31 15:52:53.309422 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:53.309400    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e\": container with ID starting with f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e not found: ID does not exist" containerID="f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"
Oct 31 15:52:53.309481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.309432    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e"} err="failed to get container status \"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e\": rpc error: code = NotFound desc = could not find container \"f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e\": container with ID starting with f267b8f87f4c5115ef088cdd3b1aba0aa784b1af522cab6e6227185f27e4904e not found: ID does not exist"
Oct 31 15:52:53.309481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.309450    4549 scope.go:122] "RemoveContainer" containerID="087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"
Oct 31 15:52:53.321741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.321719    4549 scope.go:122] "RemoveContainer" containerID="087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"
Oct 31 15:52:53.322096 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:53.322067    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a\": container with ID starting with 087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a not found: ID does not exist" containerID="087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"
Oct 31 15:52:53.322181 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.322104    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a"} err="failed to get container status \"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a\": rpc error: code = NotFound desc = could not find container \"087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a\": container with ID starting with 087b71d594be11aba2939fb9008134a15e989f80dbfca5c34adc2e11cedda98a not found: ID does not exist"
Oct 31 15:52:53.422794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.422758    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfe341c-8db2-4a11-8d3f-42e553dda630" path="/var/lib/kubelet/pods/3cfe341c-8db2-4a11-8d3f-42e553dda630/volumes"
Oct 31 15:52:53.457885 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.457848    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:52:53.508212 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.508178    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73b8d8-86758fc447-gl7zk"]
Oct 31 15:52:53.614862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.614777    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:52:53.618800 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:53.618775    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73b8d8-54bf6db8f-tpk4k"]
Oct 31 15:52:54.786229 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:54.786188    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerName="payload-processing" containerID="cri-o://8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8" gracePeriod=25
Oct 31 15:52:54.834513 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:52:54.834491    4549 logging.go:55] [core] [Channel #1805 SubChannel #1806]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.173:50052: connect: connection refused"
Oct 31 15:52:55.048540 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.048466    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:52:55.131695 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.131665    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\") pod \"dcbe4a36-e1be-4db2-9479-ae15d162e805\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") "
Oct 31 15:52:55.131911 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.131737    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\" (UniqueName: \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\") pod \"dcbe4a36-e1be-4db2-9479-ae15d162e805\" (UID: \"dcbe4a36-e1be-4db2-9479-ae15d162e805\") "
Oct 31 15:52:55.132409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.132087    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume" pod "dcbe4a36-e1be-4db2-9479-ae15d162e805" (UID: "dcbe4a36-e1be-4db2-9479-ae15d162e805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:52:55.133121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.133094    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe4a36-e1be-4db2-9479-ae15d162e805-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:55.135026 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.134994    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt" pod "dcbe4a36-e1be-4db2-9479-ae15d162e805" (UID: "dcbe4a36-e1be-4db2-9479-ae15d162e805"). InnerVolumeSpecName "kube-api-access-krrwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:52:55.234678 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.234648    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrwt\" (UniqueName: \"kubernetes.io/projected/dcbe4a36-e1be-4db2-9479-ae15d162e805-kube-api-access-krrwt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:52:55.314058 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.313958    4549 generic.go:356] "Generic (PLEG): container finished" podID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerID="8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8" exitCode=0
Oct 31 15:52:55.314058 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.314033    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"
Oct 31 15:52:55.314240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.314045    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" event={"ID":"dcbe4a36-e1be-4db2-9479-ae15d162e805","Type":"ContainerDied","Data":"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"}
Oct 31 15:52:55.314240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.314136    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" event={"ID":"dcbe4a36-e1be-4db2-9479-ae15d162e805","Type":"ContainerDied","Data":"80a5e8273faac1af6afff9819755070fec1650a77243d89f87e4d012d0571159"}
Oct 31 15:52:55.314240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.314188    4549 scope.go:122] "RemoveContainer" containerID="8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"
Oct 31 15:52:55.325526 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.325505    4549 scope.go:122] "RemoveContainer" containerID="8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"
Oct 31 15:52:55.325881 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:52:55.325859    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8\": container with ID starting with 8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8 not found: ID does not exist" containerID="8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"
Oct 31 15:52:55.325993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.325888    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8"} err="failed to get container status \"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8\": rpc error: code = NotFound desc = could not find container \"8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8\": container with ID starting with 8bab5f5caebf15898de154f1742e2d2189b9e65662b0f0e143473c8735ee61f8 not found: ID does not exist"
Oct 31 15:52:55.419388 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.419352    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719f3fc4-7f98-451d-93df-431cf106fb63" path="/var/lib/kubelet/pods/719f3fc4-7f98-451d-93df-431cf106fb63/volumes"
Oct 31 15:52:55.419776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.419762    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1911d49-ec44-451b-ad14-27ba00f49151" path="/var/lib/kubelet/pods/b1911d49-ec44-451b-ad14-27ba00f49151/volumes"
Oct 31 15:52:55.477645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.477077    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:52:55.480369 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.480337    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g"]
Oct 31 15:52:55.834971 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:55.834929    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4c9a78-5565cbcf75-lkh4g" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Oct 31 15:52:57.481356 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:52:57.479227    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" path="/var/lib/kubelet/pods/dcbe4a36-e1be-4db2-9479-ae15d162e805/volumes"
Oct 31 15:53:07.013173 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:53:07.013128    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:53:37.014620 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:53:37.014565    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:53:59.270993 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.270957    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:53:59.271480 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.271208    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" podUID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" containerName="maas-api" containerID="cri-o://c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46" gracePeriod=30
Oct 31 15:53:59.604345 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.604317    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:53:59.663915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.663883    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\") pod \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") "
Oct 31 15:53:59.664763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.664741    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\" (UniqueName: \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\") pod \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") "
Oct 31 15:53:59.665317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.665299    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\") pod \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") "
Oct 31 15:53:59.665535 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.665522    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\") pod \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\" (UID: \"9ccba9db-066d-46bd-adb3-9a841dc5fb78\") "
Oct 31 15:53:59.666517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.664627    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca" pod "9ccba9db-066d-46bd-adb3-9a841dc5fb78" (UID: "9ccba9db-066d-46bd-adb3-9a841dc5fb78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:53:59.676655 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.676598    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq" pod "9ccba9db-066d-46bd-adb3-9a841dc5fb78" (UID: "9ccba9db-066d-46bd-adb3-9a841dc5fb78"). InnerVolumeSpecName "kube-api-access-mg6jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:53:59.677936 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.677893    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs" pod "9ccba9db-066d-46bd-adb3-9a841dc5fb78" (UID: "9ccba9db-066d-46bd-adb3-9a841dc5fb78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:53:59.678194 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.678146    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls" pod "9ccba9db-066d-46bd-adb3-9a841dc5fb78" (UID: "9ccba9db-066d-46bd-adb3-9a841dc5fb78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:53:59.767772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.767731    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6jq\" (UniqueName: \"kubernetes.io/projected/9ccba9db-066d-46bd-adb3-9a841dc5fb78-kube-api-access-mg6jq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:53:59.767772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.767763    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:53:59.767772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.767773    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ccba9db-066d-46bd-adb3-9a841dc5fb78-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:53:59.767772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.767782    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ccba9db-066d-46bd-adb3-9a841dc5fb78-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:53:59.839635 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.839526    4549 generic.go:356] "Generic (PLEG): container finished" podID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" containerID="c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46" exitCode=0
Oct 31 15:53:59.839635 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.839607    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"
Oct 31 15:53:59.839881 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.839630    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" event={"ID":"9ccba9db-066d-46bd-adb3-9a841dc5fb78","Type":"ContainerDied","Data":"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"}
Oct 31 15:53:59.839881 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.839691    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l" event={"ID":"9ccba9db-066d-46bd-adb3-9a841dc5fb78","Type":"ContainerDied","Data":"83df07e1e9a60e04d871d54df71055ec65a47e187cb6db317fe6a2ff4c65c112"}
Oct 31 15:53:59.839881 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.839719    4549 scope.go:122] "RemoveContainer" containerID="c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"
Oct 31 15:53:59.852548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.852517    4549 scope.go:122] "RemoveContainer" containerID="c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"
Oct 31 15:53:59.853131 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:53:59.853060    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46\": container with ID starting with c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46 not found: ID does not exist" containerID="c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"
Oct 31 15:53:59.853131 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:53:59.853103    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46"} err="failed to get container status \"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46\": rpc error: code = NotFound desc = could not find container \"c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46\": container with ID starting with c6bdc3f185761e1324bfd3ee81631058a40db38ac3b7972a941ab2af40d23f46 not found: ID does not exist"
Oct 31 15:54:00.003811 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:00.003772    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:54:00.009027 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:00.008998    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-73b8d8-56c487fb5-s766l"]
Oct 31 15:54:01.419468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:01.419426    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" path="/var/lib/kubelet/pods/9ccba9db-066d-46bd-adb3-9a841dc5fb78/volumes"
Oct 31 15:54:05.080126 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.080091    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:54:05.080622 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.080280    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" podUID="4c3391d6-86f4-4e10-8f48-03a8dc186414" containerName="maas-api" containerID="cri-o://9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e" gracePeriod=30
Oct 31 15:54:05.372516 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.372486    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:54:05.431869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.431842    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\") pod \"4c3391d6-86f4-4e10-8f48-03a8dc186414\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") "
Oct 31 15:54:05.432075 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.431928    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") pod \"4c3391d6-86f4-4e10-8f48-03a8dc186414\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") "
Oct 31 15:54:05.432147 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.432076    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\" (UniqueName: \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\") pod \"4c3391d6-86f4-4e10-8f48-03a8dc186414\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") "
Oct 31 15:54:05.432234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.432204    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\") pod \"4c3391d6-86f4-4e10-8f48-03a8dc186414\" (UID: \"4c3391d6-86f4-4e10-8f48-03a8dc186414\") "
Oct 31 15:54:05.432628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.432596    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca" pod "4c3391d6-86f4-4e10-8f48-03a8dc186414" (UID: "4c3391d6-86f4-4e10-8f48-03a8dc186414"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:54:05.434536 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.434504    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs" pod "4c3391d6-86f4-4e10-8f48-03a8dc186414" (UID: "4c3391d6-86f4-4e10-8f48-03a8dc186414"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:54:05.434765 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.434741    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls" pod "4c3391d6-86f4-4e10-8f48-03a8dc186414" (UID: "4c3391d6-86f4-4e10-8f48-03a8dc186414"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:54:05.434838 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.434779    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc" pod "4c3391d6-86f4-4e10-8f48-03a8dc186414" (UID: "4c3391d6-86f4-4e10-8f48-03a8dc186414"). InnerVolumeSpecName "kube-api-access-zjtcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:54:05.534399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.534362    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtcc\" (UniqueName: \"kubernetes.io/projected/4c3391d6-86f4-4e10-8f48-03a8dc186414-kube-api-access-zjtcc\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:05.534399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.534387    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c3391d6-86f4-4e10-8f48-03a8dc186414-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:05.534399 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.534397    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:05.534744 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.534413    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c3391d6-86f4-4e10-8f48-03a8dc186414-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:05.884968 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.884934    4549 generic.go:356] "Generic (PLEG): container finished" podID="4c3391d6-86f4-4e10-8f48-03a8dc186414" containerID="9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e" exitCode=0
Oct 31 15:54:05.885133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.884989    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"
Oct 31 15:54:05.885133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.885013    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" event={"ID":"4c3391d6-86f4-4e10-8f48-03a8dc186414","Type":"ContainerDied","Data":"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"}
Oct 31 15:54:05.885133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.885067    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr" event={"ID":"4c3391d6-86f4-4e10-8f48-03a8dc186414","Type":"ContainerDied","Data":"f59e993b91836b3ee1f1efdb375f3b3ec80db9868d2ca76dfcbca4ef353917c1"}
Oct 31 15:54:05.885133 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.885091    4549 scope.go:122] "RemoveContainer" containerID="9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"
Oct 31 15:54:05.897348 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.897328    4549 scope.go:122] "RemoveContainer" containerID="9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"
Oct 31 15:54:05.897644 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:05.897624    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e\": container with ID starting with 9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e not found: ID does not exist" containerID="9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"
Oct 31 15:54:05.897754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:05.897661    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e"} err="failed to get container status \"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e\": rpc error: code = NotFound desc = could not find container \"9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e\": container with ID starting with 9fd7139777abebf0611f960824f9ec3f899944f95f1dbb5d70b3f86eb327858e not found: ID does not exist"
Oct 31 15:54:06.054079 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:06.054015    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:54:06.059177 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:06.059145    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c4840c-657c8f89d7-57pqr"]
Oct 31 15:54:07.013527 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:07.013483    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:54:07.323213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:07.323175    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:54:07.323447 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:07.323394    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" containerID="cri-o://101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4" gracePeriod=30
Oct 31 15:54:07.425858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:07.425825    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3391d6-86f4-4e10-8f48-03a8dc186414" path="/var/lib/kubelet/pods/4c3391d6-86f4-4e10-8f48-03a8dc186414/volumes"
Oct 31 15:54:09.138716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:09.138683    4549 patch_prober.go:29] interesting pod/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 15:54:09.139162 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:09.138746    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 15:54:12.609120 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.609089    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:54:12.649125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649088    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649140    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649212    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649266    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649307    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649451 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649413    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649507 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649463    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649502    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649516    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:54:12.649565 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649535    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\") pod \"a2e8519b-7412-4e11-9e40-1da6e7d43296\" (UID: \"a2e8519b-7412-4e11-9e40-1da6e7d43296\") "
Oct 31 15:54:12.649965 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.649899    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:54:12.650110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.650036    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.650110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.650071    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.650262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.650241    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:54:12.650404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.650264    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:54:12.650404 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.650363    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:54:12.651674 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.651653    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:54:12.651781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.651752    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:54:12.652099 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.652071    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:54:12.652247 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.652197    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc" pod "a2e8519b-7412-4e11-9e40-1da6e7d43296" (UID: "a2e8519b-7412-4e11-9e40-1da6e7d43296"). InnerVolumeSpecName "kube-api-access-m5nrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:54:12.751115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751083    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751110    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751115 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751122    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751130    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751140    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5nrc\" (UniqueName: \"kubernetes.io/projected/a2e8519b-7412-4e11-9e40-1da6e7d43296-kube-api-access-m5nrc\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751149    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e8519b-7412-4e11-9e40-1da6e7d43296-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.751367 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.751157    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e8519b-7412-4e11-9e40-1da6e7d43296-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:12.943322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.943238    4549 generic.go:356] "Generic (PLEG): container finished" podID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerID="101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4" exitCode=0
Oct 31 15:54:12.943322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.943293    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"
Oct 31 15:54:12.943510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.943332    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerDied","Data":"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"}
Oct 31 15:54:12.943510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.943403    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs" event={"ID":"a2e8519b-7412-4e11-9e40-1da6e7d43296","Type":"ContainerDied","Data":"248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f"}
Oct 31 15:54:12.943510 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.943438    4549 scope.go:122] "RemoveContainer" containerID="101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"
Oct 31 15:54:12.955973 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.955950    4549 scope.go:122] "RemoveContainer" containerID="9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"
Oct 31 15:54:12.966637 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.966615    4549 scope.go:122] "RemoveContainer" containerID="101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"
Oct 31 15:54:12.967339 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:12.967317    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4\": container with ID starting with 101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4 not found: ID does not exist" containerID="101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"
Oct 31 15:54:12.967422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.967350    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4"} err="failed to get container status \"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4\": rpc error: code = NotFound desc = could not find container \"101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4\": container with ID starting with 101b86e22340637988c6ab9bf75ea3581a61a87e7eb162590caa5643f55c01f4 not found: ID does not exist"
Oct 31 15:54:12.967422 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.967376    4549 scope.go:122] "RemoveContainer" containerID="9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"
Oct 31 15:54:12.967800 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:12.967772    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b\": container with ID starting with 9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b not found: ID does not exist" containerID="9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"
Oct 31 15:54:12.967891 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:12.967810    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b"} err="failed to get container status \"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b\": rpc error: code = NotFound desc = could not find container \"9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b\": container with ID starting with 9653aa9c4857989ff641480935dc7b99170823abffff52014131b76984a4c52b not found: ID does not exist"
Oct 31 15:54:13.015084 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:13.015040    4549 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e8519b_7412_4e11_9e40_1da6e7d43296.slice/crio-248200c5a90efe59438670769b5df2a251e33fd12101e19b78ebacd39d25d82f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e8519b_7412_4e11_9e40_1da6e7d43296.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 15:54:13.108045 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:13.107996    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:54:13.110876 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:13.110854    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c4840c-openshift-default-d7bf9879-gznqs"]
Oct 31 15:54:13.421531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:13.421492    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" path="/var/lib/kubelet/pods/a2e8519b-7412-4e11-9e40-1da6e7d43296/volumes"
Oct 31 15:54:37.012704 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:54:37.012666    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:54:44.876141 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:44.876105    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:54:44.876711 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:44.876299    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" podUID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" containerName="maas-api" containerID="cri-o://2c3639220cc9b9e0b1d7d942692cdc49dd3732e9f50e9c52381e7abdf9b8cf88" gracePeriod=30
Oct 31 15:54:45.160192 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.160107    4549 generic.go:356] "Generic (PLEG): container finished" podID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" containerID="2c3639220cc9b9e0b1d7d942692cdc49dd3732e9f50e9c52381e7abdf9b8cf88" exitCode=0
Oct 31 15:54:45.160372 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.160204    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" event={"ID":"87244d9d-d1bd-4c60-a8b4-5719dc869b4a","Type":"ContainerDied","Data":"2c3639220cc9b9e0b1d7d942692cdc49dd3732e9f50e9c52381e7abdf9b8cf88"}
Oct 31 15:54:45.201690 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.201666    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:54:45.224370 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.224346    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") pod \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") "
Oct 31 15:54:45.224542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.224444    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\" (UniqueName: \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\") pod \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") "
Oct 31 15:54:45.224542 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.224516    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\") pod \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") "
Oct 31 15:54:45.224688 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.224554    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\") pod \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\" (UID: \"87244d9d-d1bd-4c60-a8b4-5719dc869b4a\") "
Oct 31 15:54:45.225109 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.224874    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca" pod "87244d9d-d1bd-4c60-a8b4-5719dc869b4a" (UID: "87244d9d-d1bd-4c60-a8b4-5719dc869b4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:54:45.225233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.225109    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:45.227200 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.227142    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls" pod "87244d9d-d1bd-4c60-a8b4-5719dc869b4a" (UID: "87244d9d-d1bd-4c60-a8b4-5719dc869b4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:54:45.227548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.227517    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs" pod "87244d9d-d1bd-4c60-a8b4-5719dc869b4a" (UID: "87244d9d-d1bd-4c60-a8b4-5719dc869b4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:54:45.227942 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.227922    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt" pod "87244d9d-d1bd-4c60-a8b4-5719dc869b4a" (UID: "87244d9d-d1bd-4c60-a8b4-5719dc869b4a"). InnerVolumeSpecName "kube-api-access-q5dtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:54:45.332509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.328748    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:45.332509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.328780    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dtt\" (UniqueName: \"kubernetes.io/projected/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-kube-api-access-q5dtt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:45.332509 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:45.328795    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87244d9d-d1bd-4c60-a8b4-5719dc869b4a-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:54:46.168181 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:46.168142    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"
Oct 31 15:54:46.168706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:46.168182    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8" event={"ID":"87244d9d-d1bd-4c60-a8b4-5719dc869b4a","Type":"ContainerDied","Data":"b1d5a7a2e296dd69e79d0c7429602a3288a034bb8c92a7cb65c152d12b870c0a"}
Oct 31 15:54:46.168706 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:46.168254    4549 scope.go:122] "RemoveContainer" containerID="2c3639220cc9b9e0b1d7d942692cdc49dd3732e9f50e9c52381e7abdf9b8cf88"
Oct 31 15:54:46.332065 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:46.332028    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:54:46.334904 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:46.334882    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-883697-584ddcb8d8-bdkj8"]
Oct 31 15:54:47.419261 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:54:47.419222    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" path="/var/lib/kubelet/pods/87244d9d-d1bd-4c60-a8b4-5719dc869b4a/volumes"
Oct 31 15:55:07.012668 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:55:07.012604    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:55:26.866758 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.866717    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:55:26.867394 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867379    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" containerName="payload-processing"
Oct 31 15:55:26.867440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867397    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" containerName="payload-processing"
Oct 31 15:55:26.867440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867411    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" containerName="payload-pre-processing"
Oct 31 15:55:26.867440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867419    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" containerName="payload-pre-processing"
Oct 31 15:55:26.867440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867438    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b1911d49-ec44-451b-ad14-27ba00f49151" containerName="payload-pre-processing"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867446    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1911d49-ec44-451b-ad14-27ba00f49151" containerName="payload-pre-processing"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867461    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867468    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867478    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4c3391d6-86f4-4e10-8f48-03a8dc186414" containerName="maas-api"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867486    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3391d6-86f4-4e10-8f48-03a8dc186414" containerName="maas-api"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867496    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867503    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867515    4549 cpu_manager.go:414] "containerMap: removing container" podUID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" containerName="maas-api"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867521    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" containerName="maas-api"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867542    4549 cpu_manager.go:414] "containerMap: removing container" podUID="719f3fc4-7f98-451d-93df-431cf106fb63" containerName="payload-processing"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867549    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719f3fc4-7f98-451d-93df-431cf106fb63" containerName="payload-processing"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867561    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerName="payload-processing"
Oct 31 15:55:26.867593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867583    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerName="payload-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867603    4549 cpu_manager.go:414] "containerMap: removing container" podUID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867616    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867637    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3cfe341c-8db2-4a11-8d3f-42e553dda630" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867644    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfe341c-8db2-4a11-8d3f-42e553dda630" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867745    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfe341c-8db2-4a11-8d3f-42e553dda630" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867760    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d00d0ce-58fa-445e-9b1a-83ea6414eacf" containerName="payload-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867770    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fcec7ec-d23e-4d2c-b9f1-6a42a7091b5d" containerName="payload-pre-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867780    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1911d49-ec44-451b-ad14-27ba00f49151" containerName="payload-pre-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867790    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867800    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3391d6-86f4-4e10-8f48-03a8dc186414" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867808    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e8519b-7412-4e11-9e40-1da6e7d43296" containerName="istio-proxy"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867816    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="87244d9d-d1bd-4c60-a8b4-5719dc869b4a" containerName="maas-api"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867827    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="719f3fc4-7f98-451d-93df-431cf106fb63" containerName="payload-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867836    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe4a36-e1be-4db2-9479-ae15d162e805" containerName="payload-processing"
Oct 31 15:55:26.868023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.867846    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ccba9db-066d-46bd-adb3-9a841dc5fb78" containerName="maas-api"
Oct 31 15:55:26.871490 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.871473    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:26.923304 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.923278    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9d627707-serving-cert\""
Oct 31 15:55:26.978110 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:26.978069    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:55:27.021185 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.021151    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.021500 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.021467    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.021669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.021539    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.021669 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.021603    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsv2\" (UniqueName: \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.123214 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.123132    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsv2\" (UniqueName: \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.123383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.123290    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.123383 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.123363    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.123517 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.123414    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.123517 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:55:27.123464    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-serving-cert: secret "maas-api-e2e-mt-9d627707-serving-cert" not found
Oct 31 15:55:27.123665 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:55:27.123547    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls podName:41bb01e6-c36d-4973-835d-567a865819e3 nodeName:}" failed. No retries permitted until 2026-10-31 15:55:27.623523746 +0000 UTC m=+4168.349804715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls") pod "maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" (UID: "41bb01e6-c36d-4973-835d-567a865819e3") : secret "maas-api-e2e-mt-9d627707-serving-cert" not found
Oct 31 15:55:27.124083 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.124061    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.126346 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.126322    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.180453 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.180413    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsv2\" (UniqueName: \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.629762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.629727    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.632338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.632310    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") pod \"maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:27.784659 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:27.784623    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:28.160787 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:28.160753    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:55:28.161545 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:55:28.161516    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bb01e6_c36d_4973_835d_567a865819e3.slice/crio-45fe22a7f74be5dc5981fb0f30923ad6a623470a47967bba2e3a1d581173cb82 WatchSource:0}: Error finding container 45fe22a7f74be5dc5981fb0f30923ad6a623470a47967bba2e3a1d581173cb82: Status 404 returned error can't find the container with id 45fe22a7f74be5dc5981fb0f30923ad6a623470a47967bba2e3a1d581173cb82
Oct 31 15:55:28.163044 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:28.163024    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 15:55:28.473320 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:28.473219    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" event={"ID":"41bb01e6-c36d-4973-835d-567a865819e3","Type":"ContainerStarted","Data":"45fe22a7f74be5dc5981fb0f30923ad6a623470a47967bba2e3a1d581173cb82"}
Oct 31 15:55:29.481596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:29.481469    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" event={"ID":"41bb01e6-c36d-4973-835d-567a865819e3","Type":"ContainerStarted","Data":"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"}
Oct 31 15:55:29.482073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:29.481695    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:29.614987 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:29.614931    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" podStartSLOduration=2.552067126 podStartE2EDuration="3.614917539s" totalImagesPullingTime="1.062850413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:55:29.613293244 +0000 UTC m=+4170.339574223" watchObservedRunningTime="2026-10-31 15:55:29.614917539 +0000 UTC m=+4170.341198508"
Oct 31 15:55:35.493994 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:55:35.493966    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:55:37.015865 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:55:37.015822    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:56:03.158151 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:03.158123    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:56:03.305719 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:03.305691    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 15:56:03.766188 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:03.766158    4549 scope.go:122] "RemoveContainer" containerID="299bf577262d1021f162c1969d9bff909edbd57207c1eb26b124668f2846d141"
Oct 31 15:56:07.016402 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:56:07.016366    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:56:15.075822 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:15.075786    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:56:15.076688 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:15.076652    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy" containerID="cri-o://f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81" gracePeriod=30
Oct 31 15:56:20.398398 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.398365    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:56:20.470121 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470079    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470223    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470331 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470254    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470341    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470374    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470410    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470511 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470450    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.470892 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470456    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:56:20.470991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470742    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:56:20.470991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470978    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:56:20.471136 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.470879    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.471273 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.471256    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\") pod \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\" (UID: \"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5\") "
Oct 31 15:56:20.471620 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.471315    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:56:20.471745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.471544    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:56:20.472267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.472244    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.472390 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.472377    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.472481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.472469    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.472595 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.472564    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.472698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.472685    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.473471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.473343    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:56:20.473605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.473527    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "kube-api-access-7ns2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:56:20.474431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.474394    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:56:20.474950 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.474916    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy" pod "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" (UID: "0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:56:20.573869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.573837    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.573869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.573868    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.573869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.573884    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.574159 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.573897    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ns2p\" (UniqueName: \"kubernetes.io/projected/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5-kube-api-access-7ns2p\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:56:20.872552 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.872477    4549 generic.go:356] "Generic (PLEG): container finished" podID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerID="f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81" exitCode=0
Oct 31 15:56:20.872826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.872558    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerDied","Data":"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"}
Oct 31 15:56:20.872826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.872691    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl" event={"ID":"0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5","Type":"ContainerDied","Data":"823004ab86d2efc5ac14207b32962b911c97c4bdb082383a405089e03e0f99c7"}
Oct 31 15:56:20.872826 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.872742    4549 scope.go:122] "RemoveContainer" containerID="f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"
Oct 31 15:56:20.873298 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.873270    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"
Oct 31 15:56:20.896969 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.896941    4549 scope.go:122] "RemoveContainer" containerID="240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"
Oct 31 15:56:20.910031 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.910000    4549 scope.go:122] "RemoveContainer" containerID="f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"
Oct 31 15:56:20.910460 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:56:20.910426    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81\": container with ID starting with f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81 not found: ID does not exist" containerID="f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"
Oct 31 15:56:20.910627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.910472    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81"} err="failed to get container status \"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81\": rpc error: code = NotFound desc = could not find container \"f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81\": container with ID starting with f885525e06a867746a0dee6a128a1921d1e8289a4ddd8ec18919c32902c58c81 not found: ID does not exist"
Oct 31 15:56:20.910627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.910502    4549 scope.go:122] "RemoveContainer" containerID="240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"
Oct 31 15:56:20.911064 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:56:20.910982    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762\": container with ID starting with 240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762 not found: ID does not exist" containerID="240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"
Oct 31 15:56:20.911064 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:20.911023    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762"} err="failed to get container status \"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762\": rpc error: code = NotFound desc = could not find container \"240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762\": container with ID starting with 240ee83a3c78d92a9facb0a7346a785b97b9a0bd93a84c3b3ca6c7e061841762 not found: ID does not exist"
Oct 31 15:56:21.045842 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:21.045810    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:56:21.050797 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:21.050770    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cf95c9-openshift-default-f85b7dc8c-nn7zl"]
Oct 31 15:56:21.469709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:21.469670    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" path="/var/lib/kubelet/pods/0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5/volumes"
Oct 31 15:56:23.148596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:23.147702    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-lbl5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:56:23.148596 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:23.147791    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:56:23.377778 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:23.377718    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 15:56:23.377970 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:23.377794    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 15:56:24.378548 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.378512    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 15:56:24.379262 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.379243    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.379312 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.379267    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.379441 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.379429    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.379485 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.379452    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.384379 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.384349    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.437233 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.437203    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 15:56:24.483294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.483267    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-31dff8\""
Oct 31 15:56:24.487915 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.487889    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkvt\" (UniqueName: \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.488080 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.488060    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.539267 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.539223    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 15:56:24.540318 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.540280    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.543023 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.542994    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdaaa87-ee5e-4ce2-98ed-a91c7b22d4a5" containerName="istio-proxy"
Oct 31 15:56:24.573819 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.573785    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.589060 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.589032    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkvt\" (UniqueName: \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.589232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.589157    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.589232 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.589195    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x72d9\" (UniqueName: \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.589326 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.589296    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.590041 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.590020    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.596886 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.596860    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 15:56:24.628315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.628286    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-31dff8-dockercfg-jvnhc\""
Oct 31 15:56:24.645785 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.645716    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkvt\" (UniqueName: \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\") pod \"payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.690093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.690056    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x72d9\" (UniqueName: \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.690317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.690139    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.690907 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.690884    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.697879 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.697857    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:24.746477 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.746446    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x72d9\" (UniqueName: \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\") pod \"payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:24.887912 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:24.887876    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:25.078696 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.076610    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 15:56:25.264951 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.264926    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 15:56:25.265961 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:56:25.265939    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b09986_6611_4d04_9706_2b4c291c9720.slice/crio-30ceea3bd539ba0ff27c3abbc6a6dd082c295f0a90465ee0d71be8f18a80d956 WatchSource:0}: Error finding container 30ceea3bd539ba0ff27c3abbc6a6dd082c295f0a90465ee0d71be8f18a80d956: Status 404 returned error can't find the container with id 30ceea3bd539ba0ff27c3abbc6a6dd082c295f0a90465ee0d71be8f18a80d956
Oct 31 15:56:25.914722 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.914685    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" event={"ID":"c7b09986-6611-4d04-9706-2b4c291c9720","Type":"ContainerStarted","Data":"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"}
Oct 31 15:56:25.915209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.914745    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:25.915209 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.914768    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" event={"ID":"c7b09986-6611-4d04-9706-2b4c291c9720","Type":"ContainerStarted","Data":"30ceea3bd539ba0ff27c3abbc6a6dd082c295f0a90465ee0d71be8f18a80d956"}
Oct 31 15:56:25.916468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.916446    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" event={"ID":"9a398640-2695-466c-a576-8d241f12838f","Type":"ContainerStarted","Data":"c35b42a95a74622867ecf86f5e904f9e9f65d5b83bddcc01b9cc26f1d45d2dd3"}
Oct 31 15:56:25.916593 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.916490    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" event={"ID":"9a398640-2695-466c-a576-8d241f12838f","Type":"ContainerStarted","Data":"1de00a8909b9bb1fb104dff3cdb1de4aeeca5ec5083ec9fb81a109a44e6b386d"}
Oct 31 15:56:25.916645 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:25.916600    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:26.029183 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:26.029109    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" podStartSLOduration=2.029089919 podStartE2EDuration="2.029089919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:56:26.025984935 +0000 UTC m=+4226.752265911" watchObservedRunningTime="2026-10-31 15:56:26.029089919 +0000 UTC m=+4226.755370900"
Oct 31 15:56:26.155614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:26.155419    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" podStartSLOduration=2.155396091 podStartE2EDuration="2.155396091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:56:26.150379978 +0000 UTC m=+4226.876660957" watchObservedRunningTime="2026-10-31 15:56:26.155396091 +0000 UTC m=+4226.881677070"
Oct 31 15:56:36.926688 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:36.926657    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 15:56:36.927297 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:36.926722    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 15:56:37.013242 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:56:37.013205    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:56:48.775476 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:48.774694    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:56:48.880900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:48.880864    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:56:53.775712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:53.775672    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing" containerID="cri-o://30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d" gracePeriod=25
Oct 31 15:56:53.882149 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:53.882106    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing" containerID="cri-o://d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da" gracePeriod=25
Oct 31 15:56:59.504651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:59.504610    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 15:56:59.504651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:56:59.504632    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 15:57:07.013094 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:07.013051    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:57:09.505633 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:09.505565    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 15:57:09.506230 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:09.505714    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 15:57:14.046781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.046495    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:57:14.047433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.047164    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\" (UniqueName: \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\") pod \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") "
Oct 31 15:57:14.047433 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.047262    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\") pod \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\" (UID: \"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3\") "
Oct 31 15:57:14.047932 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.047775    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume" pod "9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" (UID: "9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:57:14.049777 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.049746    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp" pod "9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" (UID: "9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3"). InnerVolumeSpecName "kube-api-access-cfhkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:57:14.135047 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.135022    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:57:14.148134 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.148114    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\" (UniqueName: \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\") pod \"65d3a5b2-3c24-4145-b539-385e6dc4d847\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") "
Oct 31 15:57:14.148249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.148216    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\") pod \"65d3a5b2-3c24-4145-b539-385e6dc4d847\" (UID: \"65d3a5b2-3c24-4145-b539-385e6dc4d847\") "
Oct 31 15:57:14.148461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.148446    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:57:14.148461 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.148461    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhkp\" (UniqueName: \"kubernetes.io/projected/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3-kube-api-access-cfhkp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:57:14.148627 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.148593    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume" pod "65d3a5b2-3c24-4145-b539-385e6dc4d847" (UID: "65d3a5b2-3c24-4145-b539-385e6dc4d847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:57:14.150287 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.150269    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn" pod "65d3a5b2-3c24-4145-b539-385e6dc4d847" (UID: "65d3a5b2-3c24-4145-b539-385e6dc4d847"). InnerVolumeSpecName "kube-api-access-ffhsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:57:14.249017 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.248984    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhsn\" (UniqueName: \"kubernetes.io/projected/65d3a5b2-3c24-4145-b539-385e6dc4d847-kube-api-access-ffhsn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:57:14.249017 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.249012    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3a5b2-3c24-4145-b539-385e6dc4d847-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:57:14.304418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.304296    4549 generic.go:356] "Generic (PLEG): container finished" podID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerID="d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da" exitCode=0
Oct 31 15:57:14.304418 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.304363    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"
Oct 31 15:57:14.304698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.304426    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" event={"ID":"65d3a5b2-3c24-4145-b539-385e6dc4d847","Type":"ContainerDied","Data":"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"}
Oct 31 15:57:14.304698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.304526    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l" event={"ID":"65d3a5b2-3c24-4145-b539-385e6dc4d847","Type":"ContainerDied","Data":"e6b0ff17b02e6fed14a9b01d719470cda04a93e17bad40207c2e88a7cb3433d8"}
Oct 31 15:57:14.304698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.304560    4549 scope.go:122] "RemoveContainer" containerID="d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"
Oct 31 15:57:14.306767 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.306733    4549 generic.go:356] "Generic (PLEG): container finished" podID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerID="30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d" exitCode=0
Oct 31 15:57:14.306899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.306785    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" event={"ID":"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3","Type":"ContainerDied","Data":"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"}
Oct 31 15:57:14.306899 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.306856    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"
Oct 31 15:57:14.307021 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.306856    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z" event={"ID":"9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3","Type":"ContainerDied","Data":"1700b280fe35588a1826ce2bad67f0f626bbba22698063cd86e82db68a706004"}
Oct 31 15:57:14.322760 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.322738    4549 scope.go:122] "RemoveContainer" containerID="d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"
Oct 31 15:57:14.323338 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:14.323309    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da\": container with ID starting with d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da not found: ID does not exist" containerID="d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"
Oct 31 15:57:14.323462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.323350    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da"} err="failed to get container status \"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da\": rpc error: code = NotFound desc = could not find container \"d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da\": container with ID starting with d184f39cdfaa9897788ff043580b0a15048c3b47ccf596589ad8cc282cbca9da not found: ID does not exist"
Oct 31 15:57:14.323462 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.323376    4549 scope.go:122] "RemoveContainer" containerID="30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"
Oct 31 15:57:14.333859 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.333839    4549 scope.go:122] "RemoveContainer" containerID="30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"
Oct 31 15:57:14.334160 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:14.334137    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d\": container with ID starting with 30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d not found: ID does not exist" containerID="30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"
Oct 31 15:57:14.334279 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.334165    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d"} err="failed to get container status \"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d\": rpc error: code = NotFound desc = could not find container \"30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d\": container with ID starting with 30eb115b7290eb03b1ac2add40b37fa3f575de2e474311442d32ac0ede47377d not found: ID does not exist"
Oct 31 15:57:14.467869 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.467837    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:57:14.470531 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.470510    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-21111f-5fb5464695-cq78l"]
Oct 31 15:57:14.579108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.579025    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:57:14.584653 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:14.584620    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-21111f-69946bdfdf-tgh5z"]
Oct 31 15:57:15.419749 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:15.419718    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" path="/var/lib/kubelet/pods/65d3a5b2-3c24-4145-b539-385e6dc4d847/volumes"
Oct 31 15:57:15.420245 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:15.420121    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" path="/var/lib/kubelet/pods/9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3/volumes"
Oct 31 15:57:29.271269 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271238    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 15:57:29.271750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271736    4549 cpu_manager.go:414] "containerMap: removing container" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing"
Oct 31 15:57:29.271801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271752    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing"
Oct 31 15:57:29.271801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271775    4549 cpu_manager.go:414] "containerMap: removing container" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing"
Oct 31 15:57:29.271801 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271780    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing"
Oct 31 15:57:29.271900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271852    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df910b1-555d-4cf9-aad4-e8a7bbeaf1a3" containerName="payload-processing"
Oct 31 15:57:29.271900 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.271865    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d3a5b2-3c24-4145-b539-385e6dc4d847" containerName="payload-pre-processing"
Oct 31 15:57:29.276537 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.276518    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.332803 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.332773    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 15:57:29.377259 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.377227    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-31dff8-serving-cert\""
Oct 31 15:57:29.424663 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.424624    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.424862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.424716    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.424862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.424816    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtb7\" (UniqueName: \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.424992 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.424926    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.526488 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.526063    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtb7\" (UniqueName: \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.526724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.526484    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.526724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.526548    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.526724 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:29.526612    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-serving-cert: secret "maas-api-e2e-shared-a-w0-31dff8-serving-cert" not found
Oct 31 15:57:29.526724 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.526652    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.526724 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:29.526681    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls podName:8b9dae8a-d3e1-49f4-b540-5055b58aefd5 nodeName:}" failed. No retries permitted until 2026-10-31 15:57:30.026659731 +0000 UTC m=+4290.752940701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls") pod "maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" (UID: "8b9dae8a-d3e1-49f4-b540-5055b58aefd5") : secret "maas-api-e2e-shared-a-w0-31dff8-serving-cert" not found
Oct 31 15:57:29.527380 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.527343    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.534939 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.534889    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:29.583698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:29.583666    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtb7\" (UniqueName: \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:30.033119 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:30.033068    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:30.035941 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:30.035900    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:30.189502 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:30.189465    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:30.573088 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:30.573061    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 15:57:30.574713 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:57:30.574679    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9dae8a_d3e1_49f4_b540_5055b58aefd5.slice/crio-f1d7458f2920913a95d9ca0a3582382b25768a8f9344660725fb98bc17925b46 WatchSource:0}: Error finding container f1d7458f2920913a95d9ca0a3582382b25768a8f9344660725fb98bc17925b46: Status 404 returned error can't find the container with id f1d7458f2920913a95d9ca0a3582382b25768a8f9344660725fb98bc17925b46
Oct 31 15:57:31.423651 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:31.423604    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" event={"ID":"8b9dae8a-d3e1-49f4-b540-5055b58aefd5","Type":"ContainerStarted","Data":"f1d7458f2920913a95d9ca0a3582382b25768a8f9344660725fb98bc17925b46"}
Oct 31 15:57:32.431508 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:32.431468    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" event={"ID":"8b9dae8a-d3e1-49f4-b540-5055b58aefd5","Type":"ContainerStarted","Data":"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"}
Oct 31 15:57:32.431964 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:32.431702    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:32.555240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:32.555178    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" podStartSLOduration=2.572766766 podStartE2EDuration="3.555160417s" totalImagesPullingTime="982.393651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:57:32.552940675 +0000 UTC m=+4293.279221675" watchObservedRunningTime="2026-10-31 15:57:32.555160417 +0000 UTC m=+4293.281441397"
Oct 31 15:57:37.012957 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:57:37.012917    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:57:38.445528 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:38.445498    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 15:57:45.367698 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:45.367660    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:57:45.420064 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:45.420036    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 15:57:45.423878 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:57:45.423853    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 15:58:00.388750 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.388691    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" containerID="cri-o://69d1534ff5cb73420271f9284018ce15e08029e6e3d8babdd71888803252017f" gracePeriod=45
Oct 31 15:58:00.440452 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.440405    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main" containerID="cri-o://7c67f629e7303419331a6c34a293dca2f077b08d062837a263edf29fad244784" gracePeriod=45
Oct 31 15:58:00.444108 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.444047    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main" containerID="cri-o://63be2ff5a0e8f4f147e578fae690b4c58dd584cd60c280268d6d3901eade31ec" gracePeriod=45
Oct 31 15:58:00.553586 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.553529    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Oct 31 15:58:00.648743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.648282    4549 generic.go:356] "Generic (PLEG): container finished" podID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerID="63be2ff5a0e8f4f147e578fae690b4c58dd584cd60c280268d6d3901eade31ec" exitCode=0
Oct 31 15:58:00.648743 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.648553    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" event={"ID":"bf98a38f-d6c2-45ad-a43c-a533ed3b59be","Type":"ContainerDied","Data":"63be2ff5a0e8f4f147e578fae690b4c58dd584cd60c280268d6d3901eade31ec"}
Oct 31 15:58:00.654988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.654774    4549 generic.go:356] "Generic (PLEG): container finished" podID="3357555d-1e57-458b-89d0-f4994654cfb0" containerID="7c67f629e7303419331a6c34a293dca2f077b08d062837a263edf29fad244784" exitCode=0
Oct 31 15:58:00.654988 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.654914    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" event={"ID":"3357555d-1e57-458b-89d0-f4994654cfb0","Type":"ContainerDied","Data":"7c67f629e7303419331a6c34a293dca2f077b08d062837a263edf29fad244784"}
Oct 31 15:58:00.657860 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.657831    4549 generic.go:356] "Generic (PLEG): container finished" podID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerID="69d1534ff5cb73420271f9284018ce15e08029e6e3d8babdd71888803252017f" exitCode=0
Oct 31 15:58:00.658033 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.657867    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" event={"ID":"8b2c1027-29aa-4c9e-95f1-4fea89d70633","Type":"ContainerDied","Data":"69d1534ff5cb73420271f9284018ce15e08029e6e3d8babdd71888803252017f"}
Oct 31 15:58:00.794768 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.794740    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:58:00.798498 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.798471    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:58:00.802439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.802416    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:58:00.893113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893073    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.893306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893220    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.893306 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893258    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.893439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893319    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.893439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893369    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.893439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893385    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.893439 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893406    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\" (UniqueName: \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893442    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893482    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893518    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\" (UniqueName: \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893564    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893630    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893698    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.893730 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893726    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893770    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893828    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\") pod \"3357555d-1e57-458b-89d0-f4994654cfb0\" (UID: \"3357555d-1e57-458b-89d0-f4994654cfb0\") "
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893857    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\" (UniqueName: \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\") pod \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\" (UID: \"8b2c1027-29aa-4c9e-95f1-4fea89d70633\") "
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893856    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893887    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.894285 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893920    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\") pod \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\" (UID: \"bf98a38f-d6c2-45ad-a43c-a533ed3b59be\") "
Oct 31 15:58:00.894646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894454    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.894646 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894477    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.893881    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894236    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894337    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894431    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894670    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.894759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894712    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.895090 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.894958    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.896415 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.896385    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:00.896991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.896955    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.896991 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.896965    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.897409 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.897387    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:00.897560 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.897538    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:58:00.897772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.897747    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz" pod "bf98a38f-d6c2-45ad-a43c-a533ed3b59be" (UID: "bf98a38f-d6c2-45ad-a43c-a533ed3b59be"). InnerVolumeSpecName "kube-api-access-hl4vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:58:00.897872 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.897853    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:00.898444 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.898421    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj" pod "3357555d-1e57-458b-89d0-f4994654cfb0" (UID: "3357555d-1e57-458b-89d0-f4994654cfb0"). InnerVolumeSpecName "kube-api-access-z8lsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:58:00.898672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.898655    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94" pod "8b2c1027-29aa-4c9e-95f1-4fea89d70633" (UID: "8b2c1027-29aa-4c9e-95f1-4fea89d70633"). InnerVolumeSpecName "kube-api-access-4bg94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995451    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995484    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995498    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995510    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2c1027-29aa-4c9e-95f1-4fea89d70633-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995524    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995537    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bg94\" (UniqueName: \"kubernetes.io/projected/8b2c1027-29aa-4c9e-95f1-4fea89d70633-kube-api-access-4bg94\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995551    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995563    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995593    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995604    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3357555d-1e57-458b-89d0-f4994654cfb0-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995613    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995630    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2c1027-29aa-4c9e-95f1-4fea89d70633-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995643    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lsj\" (UniqueName: \"kubernetes.io/projected/3357555d-1e57-458b-89d0-f4994654cfb0-kube-api-access-z8lsj\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995654    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995666    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4vz\" (UniqueName: \"kubernetes.io/projected/bf98a38f-d6c2-45ad-a43c-a533ed3b59be-kube-api-access-hl4vz\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:00.995930 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:00.995678    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3357555d-1e57-458b-89d0-f4994654cfb0-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:01.666007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.665918    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"
Oct 31 15:58:01.666007 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.665948    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq" event={"ID":"3357555d-1e57-458b-89d0-f4994654cfb0","Type":"ContainerDied","Data":"229d6db40a7022d3a1b84d2aa18ae33272df3cb476f71256bc2e3781c83db9e1"}
Oct 31 15:58:01.666585 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.666013    4549 scope.go:122] "RemoveContainer" containerID="7c67f629e7303419331a6c34a293dca2f077b08d062837a263edf29fad244784"
Oct 31 15:58:01.667410 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.667393    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"
Oct 31 15:58:01.667481 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.667417    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs" event={"ID":"8b2c1027-29aa-4c9e-95f1-4fea89d70633","Type":"ContainerDied","Data":"00b55f3fa8bc25b2c3d801e51ddeade8c2415bde1e6bd2a91ecb5f09ba78eab3"}
Oct 31 15:58:01.669294 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.669273    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"
Oct 31 15:58:01.669401 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.669274    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh" event={"ID":"bf98a38f-d6c2-45ad-a43c-a533ed3b59be","Type":"ContainerDied","Data":"cb3f6882a0ca60ca0c6b3572bbf364755e873736dc1b2d4d8cb9ebbf3867c8ca"}
Oct 31 15:58:01.677464 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.677444    4549 scope.go:122] "RemoveContainer" containerID="69d1534ff5cb73420271f9284018ce15e08029e6e3d8babdd71888803252017f"
Oct 31 15:58:01.686139 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.686123    4549 scope.go:122] "RemoveContainer" containerID="63be2ff5a0e8f4f147e578fae690b4c58dd584cd60c280268d6d3901eade31ec"
Oct 31 15:58:01.829862 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.829825    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 15:58:01.839628 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.839592    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/simulator-w4-57815b-kserve-5d447fb94c-r89zh"]
Oct 31 15:58:01.947465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.947389    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:58:01.950240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:01.950219    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/distinct-w4-57815b-kserve-57c65f547c-pgrqs"]
Oct 31 15:58:02.055864 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:02.055827    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 15:58:02.059831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:02.059803    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-57815b/premium-w4-57815b-kserve-5d4f6694c4-k2vdq"]
Oct 31 15:58:03.418989 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:03.418954    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" path="/var/lib/kubelet/pods/3357555d-1e57-458b-89d0-f4994654cfb0/volumes"
Oct 31 15:58:03.419443 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:03.419429    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" path="/var/lib/kubelet/pods/8b2c1027-29aa-4c9e-95f1-4fea89d70633/volumes"
Oct 31 15:58:03.419916 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:03.419901    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" path="/var/lib/kubelet/pods/bf98a38f-d6c2-45ad-a43c-a533ed3b59be/volumes"
Oct 31 15:58:07.012129 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:58:07.012090    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:58:37.017300 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:58:37.017249    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:58:47.430854 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.430767    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 15:58:47.431452 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.430978    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" podUID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" containerName="maas-api" containerID="cri-o://fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb" gracePeriod=30
Oct 31 15:58:47.734338 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.734308    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 15:58:47.876479 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.876448    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\" (UniqueName: \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\") pod \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") "
Oct 31 15:58:47.876697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.876560    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") pod \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") "
Oct 31 15:58:47.876697 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.876647    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\") pod \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") "
Oct 31 15:58:47.876849 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.876744    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\") pod \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\" (UID: \"4814f89f-7d1d-4c5c-bcbc-d3772636b7de\") "
Oct 31 15:58:47.877322 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.877290    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca" pod "4814f89f-7d1d-4c5c-bcbc-d3772636b7de" (UID: "4814f89f-7d1d-4c5c-bcbc-d3772636b7de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:58:47.879440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.879413    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls" pod "4814f89f-7d1d-4c5c-bcbc-d3772636b7de" (UID: "4814f89f-7d1d-4c5c-bcbc-d3772636b7de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:47.879977 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.879948    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs" pod "4814f89f-7d1d-4c5c-bcbc-d3772636b7de" (UID: "4814f89f-7d1d-4c5c-bcbc-d3772636b7de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:47.880095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.880064    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd" pod "4814f89f-7d1d-4c5c-bcbc-d3772636b7de" (UID: "4814f89f-7d1d-4c5c-bcbc-d3772636b7de"). InnerVolumeSpecName "kube-api-access-ptlrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:58:47.978143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.978051    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:47.978143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.978082    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptlrd\" (UniqueName: \"kubernetes.io/projected/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-kube-api-access-ptlrd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:47.978143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.978093    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:47.978143 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.978103    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4814f89f-7d1d-4c5c-bcbc-d3772636b7de-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:47.987882 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.987844    4549 generic.go:356] "Generic (PLEG): container finished" podID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" containerID="fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb" exitCode=0
Oct 31 15:58:47.988063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.987886    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"
Oct 31 15:58:47.988063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.987933    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" event={"ID":"4814f89f-7d1d-4c5c-bcbc-d3772636b7de","Type":"ContainerDied","Data":"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"}
Oct 31 15:58:47.988063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.987999    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4" event={"ID":"4814f89f-7d1d-4c5c-bcbc-d3772636b7de","Type":"ContainerDied","Data":"da890331b3ad4490ed7c7a2f84960b419edcbdedbb0bbb7e22e9aef35ffb3abd"}
Oct 31 15:58:47.988063 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.988030    4549 scope.go:122] "RemoveContainer" containerID="fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"
Oct 31 15:58:47.999296 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.999278    4549 scope.go:122] "RemoveContainer" containerID="fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"
Oct 31 15:58:47.999581 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:58:47.999544    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb\": container with ID starting with fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb not found: ID does not exist" containerID="fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"
Oct 31 15:58:47.999696 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:47.999588    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb"} err="failed to get container status \"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb\": rpc error: code = NotFound desc = could not find container \"fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb\": container with ID starting with fc28bdae23d7c99c5edf97ebe8392880ec78632580d52c1b213f991383f068eb not found: ID does not exist"
Oct 31 15:58:48.151604 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:48.151555    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 15:58:48.156763 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:48.156737    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-57815b-86df6848dd-v6km4"]
Oct 31 15:58:49.409113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409076    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 15:58:49.409605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409591    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main"
Oct 31 15:58:49.409605 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409606    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409622    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409627    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409640    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409645    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409659    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" containerName="maas-api"
Oct 31 15:58:49.409700 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409664    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" containerName="maas-api"
Oct 31 15:58:49.409890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409745    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3357555d-1e57-458b-89d0-f4994654cfb0" containerName="main"
Oct 31 15:58:49.409890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409755    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf98a38f-d6c2-45ad-a43c-a533ed3b59be" containerName="main"
Oct 31 15:58:49.409890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409762    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2c1027-29aa-4c9e-95f1-4fea89d70633" containerName="main"
Oct 31 15:58:49.409890 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.409769    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" containerName="maas-api"
Oct 31 15:58:49.415277 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.415246    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.424863 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.424838    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4814f89f-7d1d-4c5c-bcbc-d3772636b7de" path="/var/lib/kubelet/pods/4814f89f-7d1d-4c5c-bcbc-d3772636b7de/volumes"
Oct 31 15:58:49.468349 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.468318    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-140c48-openshift-default-dockercfg-gqjnt\""
Oct 31 15:58:49.493873 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.493836    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.493910    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494085 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.493944    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494098    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494150    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494225    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494270    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494321 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494308    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.494506 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.494436    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.520937 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.520903    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596038    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596106    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596134    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596169    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596189    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596220    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596257    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596281    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596376    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.596877    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.597457    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.599681 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.597725    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.600831 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.600806    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.601402 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.601368    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.605460 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.605418    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.615130 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.615100    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.658342 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.658303    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.658512 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.658307    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\") pod \"e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") " pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:49.730551 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:49.730460    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:50.172631 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:50.172600    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 15:58:50.174108 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:58:50.174083    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870ab6ea_5aee_4e04_b015_8f99ad0ce2d6.slice/crio-6b34795fdb68c613cfe49267fc10b6f5b11cf2452e643f1f7c337e5c256f958f WatchSource:0}: Error finding container 6b34795fdb68c613cfe49267fc10b6f5b11cf2452e643f1f7c337e5c256f958f: Status 404 returned error can't find the container with id 6b34795fdb68c613cfe49267fc10b6f5b11cf2452e643f1f7c337e5c256f958f
Oct 31 15:58:50.176614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:50.176564    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:58:50.176716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:50.176647    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:58:50.176716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:50.176676    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 15:58:51.010315 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:51.010279    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerStarted","Data":"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"}
Oct 31 15:58:51.010759 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:51.010344    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerStarted","Data":"6b34795fdb68c613cfe49267fc10b6f5b11cf2452e643f1f7c337e5c256f958f"}
Oct 31 15:58:51.177024 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:51.176972    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podStartSLOduration=2.176953483 podStartE2EDuration="2.176953483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:58:51.125076357 +0000 UTC m=+4371.851357367" watchObservedRunningTime="2026-10-31 15:58:51.176953483 +0000 UTC m=+4371.903234462"
Oct 31 15:58:51.731186 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:51.731151    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:51.737533 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:51.737498    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:52.017465 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:52.017441    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:52.018371 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:52.018350    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 15:58:53.435603 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.435536    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:58:53.436068 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.435819    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" podUID="41bb01e6-c36d-4973-835d-567a865819e3" containerName="maas-api" containerID="cri-o://e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe" gracePeriod=30
Oct 31 15:58:53.736889 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.736861    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:58:53.843545 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.843508    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\") pod \"41bb01e6-c36d-4973-835d-567a865819e3\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") "
Oct 31 15:58:53.843788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.843588    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") pod \"41bb01e6-c36d-4973-835d-567a865819e3\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") "
Oct 31 15:58:53.843788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.843618    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\" (UniqueName: \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\") pod \"41bb01e6-c36d-4973-835d-567a865819e3\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") "
Oct 31 15:58:53.843788 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.843656    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\") pod \"41bb01e6-c36d-4973-835d-567a865819e3\" (UID: \"41bb01e6-c36d-4973-835d-567a865819e3\") "
Oct 31 15:58:53.844084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.844056    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca" pod "41bb01e6-c36d-4973-835d-567a865819e3" (UID: "41bb01e6-c36d-4973-835d-567a865819e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:58:53.845875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.845849    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls" pod "41bb01e6-c36d-4973-835d-567a865819e3" (UID: "41bb01e6-c36d-4973-835d-567a865819e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:53.846376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.846347    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs" pod "41bb01e6-c36d-4973-835d-567a865819e3" (UID: "41bb01e6-c36d-4973-835d-567a865819e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:58:53.846468 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.846384    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2" pod "41bb01e6-c36d-4973-835d-567a865819e3" (UID: "41bb01e6-c36d-4973-835d-567a865819e3"). InnerVolumeSpecName "kube-api-access-2hsv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:58:53.945495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.945461    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:53.945495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.945489    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41bb01e6-c36d-4973-835d-567a865819e3-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:53.945495 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.945499    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hsv2\" (UniqueName: \"kubernetes.io/projected/41bb01e6-c36d-4973-835d-567a865819e3-kube-api-access-2hsv2\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:53.945761 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:53.945509    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41bb01e6-c36d-4973-835d-567a865819e3-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:58:54.032772 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.032735    4549 generic.go:356] "Generic (PLEG): container finished" podID="41bb01e6-c36d-4973-835d-567a865819e3" containerID="e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe" exitCode=0
Oct 31 15:58:54.032966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.032799    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"
Oct 31 15:58:54.032966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.032799    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" event={"ID":"41bb01e6-c36d-4973-835d-567a865819e3","Type":"ContainerDied","Data":"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"}
Oct 31 15:58:54.032966 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.032943    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz" event={"ID":"41bb01e6-c36d-4973-835d-567a865819e3","Type":"ContainerDied","Data":"45fe22a7f74be5dc5981fb0f30923ad6a623470a47967bba2e3a1d581173cb82"}
Oct 31 15:58:54.033107 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.032973    4549 scope.go:122] "RemoveContainer" containerID="e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"
Oct 31 15:58:54.045276 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.045255    4549 scope.go:122] "RemoveContainer" containerID="e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"
Oct 31 15:58:54.045602 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:58:54.045541    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe\": container with ID starting with e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe not found: ID does not exist" containerID="e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"
Oct 31 15:58:54.045709 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.045614    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe"} err="failed to get container status \"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe\": rpc error: code = NotFound desc = could not find container \"e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe\": container with ID starting with e2ca09c790fd87df5650c0a2f8230c8180df4e5ed554690e3564abddf5de52fe not found: ID does not exist"
Oct 31 15:58:54.197792 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.197758    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:58:54.200431 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:54.200404    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9d627707-5876bdbf55-ntlfz"]
Oct 31 15:58:55.420466 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:55.420432    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41bb01e6-c36d-4973-835d-567a865819e3" path="/var/lib/kubelet/pods/41bb01e6-c36d-4973-835d-567a865819e3/volumes"
Oct 31 15:58:57.629716 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.629677    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 15:58:57.631095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.631023    4549 cpu_manager.go:414] "containerMap: removing container" podUID="41bb01e6-c36d-4973-835d-567a865819e3" containerName="maas-api"
Oct 31 15:58:57.631095 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.631059    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41bb01e6-c36d-4973-835d-567a865819e3" containerName="maas-api"
Oct 31 15:58:57.631314 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.631186    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="41bb01e6-c36d-4973-835d-567a865819e3" containerName="maas-api"
Oct 31 15:58:57.636305 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.636285    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.681073 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.681044    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 15:58:57.699781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.699745    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.699947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.699884    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvqd\" (UniqueName: \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.727317 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.727288    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-140c48-dockercfg-6nrcn\""
Oct 31 15:58:57.729956 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.729936    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-140c48\""
Oct 31 15:58:57.800541 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.800505    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.800753 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.800633    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvqd\" (UniqueName: \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.801249 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.801230    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.857070 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.857039    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvqd\" (UniqueName: \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\") pod \"payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:57.955231 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:57.955147    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:58.339213 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:58.339103    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 15:58:58.339370 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:58:58.339334    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9140fa_f90a_4c7c_a07a_f5495534cae2.slice/crio-785944f0960d992d8ca8e89affdce4f1435b0d51840ce6e3019378f4397d0406 WatchSource:0}: Error finding container 785944f0960d992d8ca8e89affdce4f1435b0d51840ce6e3019378f4397d0406: Status 404 returned error can't find the container with id 785944f0960d992d8ca8e89affdce4f1435b0d51840ce6e3019378f4397d0406
Oct 31 15:58:59.071490 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:59.071450    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" event={"ID":"8e9140fa-f90a-4c7c-a07a-f5495534cae2","Type":"ContainerStarted","Data":"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"}
Oct 31 15:58:59.072030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:59.071550    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:58:59.072030 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:59.071595    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" event={"ID":"8e9140fa-f90a-4c7c-a07a-f5495534cae2","Type":"ContainerStarted","Data":"785944f0960d992d8ca8e89affdce4f1435b0d51840ce6e3019378f4397d0406"}
Oct 31 15:58:59.186851 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:58:59.186794    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" podStartSLOduration=2.186780934 podStartE2EDuration="2.186780934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:58:59.185680936 +0000 UTC m=+4379.911961917" watchObservedRunningTime="2026-10-31 15:58:59.186780934 +0000 UTC m=+4379.913061935"
Oct 31 15:59:07.013958 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:07.013913    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:59:10.081455 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:10.081427    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 15:59:24.158393 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.158346    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:59:24.159001 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.158552    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerName="maas-api" containerID="cri-o://437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf" gracePeriod=30
Oct 31 15:59:24.515865 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.515827    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:59:24.657648 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.657614    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\") pod \"0549343d-f5ed-405d-bd93-4b40bf92da51\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") "
Oct 31 15:59:24.657820 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.657764    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\" (UniqueName: \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\") pod \"0549343d-f5ed-405d-bd93-4b40bf92da51\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") "
Oct 31 15:59:24.657947 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.657930    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\") pod \"0549343d-f5ed-405d-bd93-4b40bf92da51\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") "
Oct 31 15:59:24.658021 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.657975    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") pod \"0549343d-f5ed-405d-bd93-4b40bf92da51\" (UID: \"0549343d-f5ed-405d-bd93-4b40bf92da51\") "
Oct 31 15:59:24.659602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.658316    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca" pod "0549343d-f5ed-405d-bd93-4b40bf92da51" (UID: "0549343d-f5ed-405d-bd93-4b40bf92da51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:59:24.659602 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.658451    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0549343d-f5ed-405d-bd93-4b40bf92da51-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:24.660113 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.660088    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d" pod "0549343d-f5ed-405d-bd93-4b40bf92da51" (UID: "0549343d-f5ed-405d-bd93-4b40bf92da51"). InnerVolumeSpecName "kube-api-access-4zq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:59:24.660673 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.660647    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs" pod "0549343d-f5ed-405d-bd93-4b40bf92da51" (UID: "0549343d-f5ed-405d-bd93-4b40bf92da51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:59:24.660784 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.660752    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls" pod "0549343d-f5ed-405d-bd93-4b40bf92da51" (UID: "0549343d-f5ed-405d-bd93-4b40bf92da51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 15:59:24.759858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.759770    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:24.759858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.759803    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0549343d-f5ed-405d-bd93-4b40bf92da51-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:24.759858 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:24.759816    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zq8d\" (UniqueName: \"kubernetes.io/projected/0549343d-f5ed-405d-bd93-4b40bf92da51-kube-api-access-4zq8d\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:25.240614 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.240561    4549 generic.go:356] "Generic (PLEG): container finished" podID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerID="437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf" exitCode=0
Oct 31 15:59:25.241082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.240630    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"
Oct 31 15:59:25.241082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.240644    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" event={"ID":"0549343d-f5ed-405d-bd93-4b40bf92da51","Type":"ContainerDied","Data":"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"}
Oct 31 15:59:25.241082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.240732    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" event={"ID":"0549343d-f5ed-405d-bd93-4b40bf92da51","Type":"ContainerDied","Data":"2493184e24ebab1560625e8490228adfb612f2fe5d1f730d5c0b5b9f3f2ee9d7"}
Oct 31 15:59:25.241082 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.240763    4549 scope.go:122] "RemoveContainer" containerID="437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"
Oct 31 15:59:25.251875 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.251859    4549 scope.go:122] "RemoveContainer" containerID="437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"
Oct 31 15:59:25.252166 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:25.252149    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf\": container with ID starting with 437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf not found: ID does not exist" containerID="437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"
Oct 31 15:59:25.252217 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.252174    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf"} err="failed to get container status \"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf\": rpc error: code = NotFound desc = could not find container \"437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf\": container with ID starting with 437ecb60013920c23ec68c0ba9b2a07764848e1403094e49391436247d0af3bf not found: ID does not exist"
Oct 31 15:59:25.407810 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.407467    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:59:25.454727 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:25.454694    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c"]
Oct 31 15:59:27.418904 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:27.418862    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" path="/var/lib/kubelet/pods/0549343d-f5ed-405d-bd93-4b40bf92da51/volumes"
Oct 31 15:59:27.503076 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:27.503039    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-24ca6e-c99dc8dbd-nzc7c" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.133:8443/health\": context deadline exceeded"
Oct 31 15:59:33.619240 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.619199    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 15:59:33.619781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.619764    4549 cpu_manager.go:414] "containerMap: removing container" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerName="maas-api"
Oct 31 15:59:33.619781 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.619778    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerName="maas-api"
Oct 31 15:59:33.619913 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.619898    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="0549343d-f5ed-405d-bd93-4b40bf92da51" containerName="maas-api"
Oct 31 15:59:33.624639 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.624617    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.726220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.726190    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"kube-root-ca.crt\""
Oct 31 15:59:33.726220 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.726191    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"openshift-service-ca.crt\""
Oct 31 15:59:33.726471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.726395    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"default-dockercfg-flk5h\""
Oct 31 15:59:33.726471 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.726444    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-e72453\"/\"rate-test-model-e72453-kserve-self-signed-certs\""
Oct 31 15:59:33.731332 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.731311    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 15:59:33.764125 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764087    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.764341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764148    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.764341 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764274    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.764474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764396    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txrb\" (UniqueName: \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.764474 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764425    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.764615 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.764547    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866073    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txrb\" (UniqueName: \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866117 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866120    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866218    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866261    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866307    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866420 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866348    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866662    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866683 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866672    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.866848 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.866826    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.868825 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.868797    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.869196 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.869175    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.923054 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.922977    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txrb\" (UniqueName: \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\") pod \"rate-test-model-e72453-kserve-587cdf4f7c-wdr7v\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") " pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:33.938407 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:33.938365    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:34.330741 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:34.330713    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 15:59:34.332013 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:59:34.331985    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf45538_caf4_4eca_8401_206d4b655e2f.slice/crio-97a863ba7f645ffc68630342e0e77c25dee8c5e82ed6c1271dc7cd760dd9cc47 WatchSource:0}: Error finding container 97a863ba7f645ffc68630342e0e77c25dee8c5e82ed6c1271dc7cd760dd9cc47: Status 404 returned error can't find the container with id 97a863ba7f645ffc68630342e0e77c25dee8c5e82ed6c1271dc7cd760dd9cc47
Oct 31 15:59:35.308015 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.307979    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" event={"ID":"3bf45538-caf4-4eca-8401-206d4b655e2f","Type":"ContainerStarted","Data":"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"}
Oct 31 15:59:35.308419 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.308024    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" event={"ID":"3bf45538-caf4-4eca-8401-206d4b655e2f","Type":"ContainerStarted","Data":"97a863ba7f645ffc68630342e0e77c25dee8c5e82ed6c1271dc7cd760dd9cc47"}
Oct 31 15:59:35.425532 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.425479    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" podStartSLOduration=2.425465394 podStartE2EDuration="2.425465394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:59:35.422138266 +0000 UTC m=+4416.148419248" watchObservedRunningTime="2026-10-31 15:59:35.425465394 +0000 UTC m=+4416.151746413"
Oct 31 15:59:35.639454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.639364    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:59:35.639672 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.639646    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy" containerID="cri-o://6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61" gracePeriod=30
Oct 31 15:59:35.938821 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:35.938742    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:37.012199 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:37.012162    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 15:59:40.907044 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.906984    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:59:40.961137 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961106    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961146    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961188    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961218    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961295 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961268    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961296    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961398    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961525 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961511    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961538    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:59:40.961745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961555    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\") pod \"dc2afd94-2685-40bd-9270-cbffaa35926a\" (UID: \"dc2afd94-2685-40bd-9270-cbffaa35926a\") "
Oct 31 15:59:40.961745 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961679    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 15:59:40.961918 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961790    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:59:40.961979 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961911    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:59:40.962037 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.961974    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:59:40.962093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.962059    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:40.962093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.962081    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:40.962207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.962095    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:40.962207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.962112    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2afd94-2685-40bd-9270-cbffaa35926a-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:40.963754 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.963726    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 15:59:40.964160 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.964133    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "kube-api-access-vjwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:59:40.964257 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.964165    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 15:59:40.964257 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:40.964183    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token" pod "dc2afd94-2685-40bd-9270-cbffaa35926a" (UID: "dc2afd94-2685-40bd-9270-cbffaa35926a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 15:59:41.063629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.063590    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwwc\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-kube-api-access-vjwwc\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:41.063629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.063623    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:41.063629 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.063634    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:41.063865 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.063643    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:41.063865 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.063652    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2afd94-2685-40bd-9270-cbffaa35926a-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 15:59:41.347832 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.347797    4549 generic.go:356] "Generic (PLEG): container finished" podID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerID="6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61" exitCode=0
Oct 31 15:59:41.348059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.347884    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerDied","Data":"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"}
Oct 31 15:59:41.348059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.347895    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"
Oct 31 15:59:41.348059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.347975    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk" event={"ID":"dc2afd94-2685-40bd-9270-cbffaa35926a","Type":"ContainerDied","Data":"bd2551305fb88dc641dfbdfc479a058c6adc070f2810709ed2aacbe15123a4ec"}
Oct 31 15:59:41.348059 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.347995    4549 scope.go:122] "RemoveContainer" containerID="6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"
Oct 31 15:59:41.363378 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.363354    4549 scope.go:122] "RemoveContainer" containerID="8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"
Oct 31 15:59:41.375762 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.375738    4549 scope.go:122] "RemoveContainer" containerID="6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"
Oct 31 15:59:41.376136 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:41.376116    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61\": container with ID starting with 6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61 not found: ID does not exist" containerID="6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"
Oct 31 15:59:41.376210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.376146    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61"} err="failed to get container status \"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61\": rpc error: code = NotFound desc = could not find container \"6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61\": container with ID starting with 6dbbc0f811896b1c200ed5870bce8658491848f423dd4f785100db0332ad8a61 not found: ID does not exist"
Oct 31 15:59:41.376210 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.376164    4549 scope.go:122] "RemoveContainer" containerID="8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"
Oct 31 15:59:41.376424 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:41.376408    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec\": container with ID starting with 8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec not found: ID does not exist" containerID="8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"
Oct 31 15:59:41.376486 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.376429    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec"} err="failed to get container status \"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec\": rpc error: code = NotFound desc = could not find container \"8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec\": container with ID starting with 8d263b3105fc05ae589434fafc4c8f6d35170da114ca473e7acee55cc319c1ec not found: ID does not exist"
Oct 31 15:59:41.514274 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.514238    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:59:41.563921 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:41.563887    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-24ca6e-openshift-default-78d5cdc49c-7cdjk"]
Oct 31 15:59:43.426207 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:43.426174    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" path="/var/lib/kubelet/pods/dc2afd94-2685-40bd-9270-cbffaa35926a/volumes"
Oct 31 15:59:43.939490 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:43.939453    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:43.953234 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:43.953203    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:44.378846 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:44.378816    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 15:59:50.366096 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.366065    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 15:59:50.366530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.366516    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 15:59:50.366530 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.366528    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 15:59:50.366667 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.366655    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 15:59:50.366712 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.366676    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 15:59:50.371440 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.371419    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.426830 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.426797    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 15:59:50.472084 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.472048    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-140c48-serving-cert\""
Oct 31 15:59:50.472782 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.472752    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldndq\" (UniqueName: \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.472946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.472864    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.472946 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.472930    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.473093 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.473076    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.574454 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.574411    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.574692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.574500    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.574692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.574622    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.574692 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.574678    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldndq\" (UniqueName: \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.574877 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:50.574773    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-serving-cert: secret "maas-api-e2e-shared-b-w0-140c48-serving-cert" not found
Oct 31 15:59:50.574877 ip-10-0-139-115 kubenswrapper[4549]: E1031 15:59:50.574852    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls podName:03bb4eea-047b-46ff-8ee4-835d2e67c663 nodeName:}" failed. No retries permitted until 2026-10-31 15:59:51.074830388 +0000 UTC m=+4431.801111349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls") pod "maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" (UID: "03bb4eea-047b-46ff-8ee4-835d2e67c663") : secret "maas-api-e2e-shared-b-w0-140c48-serving-cert" not found
Oct 31 15:59:50.575127 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.575106    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.577161 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.577136    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:50.633069 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:50.632987    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldndq\" (UniqueName: \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:51.081445 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:51.081407    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:51.083981 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:51.083957    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:51.285735 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:51.285698    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:51.662776 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:51.662748    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 15:59:51.665204 ip-10-0-139-115 kubenswrapper[4549]: W1031 15:59:51.665172    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bb4eea_047b_46ff_8ee4_835d2e67c663.slice/crio-e18309b0b3b429e5b479c51631e4d367352c30e54dcb332ec1bc38f185d11aa0 WatchSource:0}: Error finding container e18309b0b3b429e5b479c51631e4d367352c30e54dcb332ec1bc38f185d11aa0: Status 404 returned error can't find the container with id e18309b0b3b429e5b479c51631e4d367352c30e54dcb332ec1bc38f185d11aa0
Oct 31 15:59:52.438361 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:52.438320    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" event={"ID":"03bb4eea-047b-46ff-8ee4-835d2e67c663","Type":"ContainerStarted","Data":"e18309b0b3b429e5b479c51631e4d367352c30e54dcb332ec1bc38f185d11aa0"}
Oct 31 15:59:53.447376 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:53.447337    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" event={"ID":"03bb4eea-047b-46ff-8ee4-835d2e67c663","Type":"ContainerStarted","Data":"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"}
Oct 31 15:59:53.447910 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:53.447465    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 15:59:53.563794 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:53.563730    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" podStartSLOduration=2.698103314 podStartE2EDuration="3.563715114s" totalImagesPullingTime="865.6118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:59:53.560763356 +0000 UTC m=+4434.287044337" watchObservedRunningTime="2026-10-31 15:59:53.563715114 +0000 UTC m=+4434.289996107"
Oct 31 15:59:59.461478 ip-10-0-139-115 kubenswrapper[4549]: I1031 15:59:59.461447    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 16:00:00.194944 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.194910    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:00.195464 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.195449    4549 cpu_manager.go:414] "containerMap: removing container" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 16:00:00.195511 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.195466    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2afd94-2685-40bd-9270-cbffaa35926a" containerName="istio-proxy"
Oct 31 16:00:00.199104 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.199079    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:00.199270 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.199166    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.204170 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.204146    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.216291 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.216262    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:00.220886 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.220862    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.250763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.250734    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\""
Oct 31 16:00:00.296497 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296458    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.296728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296602    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq5wp\" (UniqueName: \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.296728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296630    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.296728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296668    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2fp\" (UniqueName: \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.296904 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296805    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.296904 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.296898    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddfw\" (UniqueName: \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.319972 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.319905    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:00.324911 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.324875    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:00.393840 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.393809    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:00.398633 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398605    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq5wp\" (UniqueName: \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.398815 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398642    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.398815 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398699    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2fp\" (UniqueName: \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.398815 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398764    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.398990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398824    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddfw\" (UniqueName: \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.398990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.398899    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.399466 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.399442    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.399466 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.399458    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.399660 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.399517    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.455529 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.455455    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddfw\" (UniqueName: \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\") pod \"maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.455717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.455557    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2fp\" (UniqueName: \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.455717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.455587    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq5wp\" (UniqueName: \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.516346 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.516314    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:00.523800 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.523758    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:00.534886 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.534861    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:00.905283 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.905244    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:00.905788 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:00:00.905759    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5313f442_4afc_4c29_a87e_a0a45573263e.slice/crio-ecf6be348f866b5d51b00fde609f6d89dc3d3158286a6e40f8cc52f44cdb3478 WatchSource:0}: Error finding container ecf6be348f866b5d51b00fde609f6d89dc3d3158286a6e40f8cc52f44cdb3478: Status 404 returned error can't find the container with id ecf6be348f866b5d51b00fde609f6d89dc3d3158286a6e40f8cc52f44cdb3478
Oct 31 16:00:00.955422 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.955397    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:00.955528 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:00:00.955471    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf768c934_6c16_487e_92e8_2ce06b6ab105.slice/crio-b437f9ea960b795a7d269747a706eab83be1ed9693b232c25e2f47358c030014 WatchSource:0}: Error finding container b437f9ea960b795a7d269747a706eab83be1ed9693b232c25e2f47358c030014: Status 404 returned error can't find the container with id b437f9ea960b795a7d269747a706eab83be1ed9693b232c25e2f47358c030014
Oct 31 16:00:00.957150 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:00:00.957126    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613ce2e0_e194_42f3_bc31_e9ba7fad4cda.slice/crio-ae05c6e63fbe61c8449017e3c22b4a8eaabecd06db8d4527d2e9985d3c6e9bcc WatchSource:0}: Error finding container ae05c6e63fbe61c8449017e3c22b4a8eaabecd06db8d4527d2e9985d3c6e9bcc: Status 404 returned error can't find the container with id ae05c6e63fbe61c8449017e3c22b4a8eaabecd06db8d4527d2e9985d3c6e9bcc
Oct 31 16:00:00.960128 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:00.960102    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:01.515922 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.515886    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerStarted","Data":"60731aa312ec8a0647ba0415fa6a27b098bd4bad2dde065fa02ad78a928dbf5b"}
Oct 31 16:00:01.516114 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.515955    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerStarted","Data":"b437f9ea960b795a7d269747a706eab83be1ed9693b232c25e2f47358c030014"}
Oct 31 16:00:01.517858 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.517822    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerStarted","Data":"03ba45fc59f4fe78daceca3be97adc9061ad6781160e9c1793e42289b9ad06bb"}
Oct 31 16:00:01.518347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.517883    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerStarted","Data":"ecf6be348f866b5d51b00fde609f6d89dc3d3158286a6e40f8cc52f44cdb3478"}
Oct 31 16:00:01.520208 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.520177    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerStarted","Data":"b0c1abbaf65649e3a600796f1c065355bc584f036b92c60aed23fecb1da6baed"}
Oct 31 16:00:01.520369 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.520232    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerStarted","Data":"ae05c6e63fbe61c8449017e3c22b4a8eaabecd06db8d4527d2e9985d3c6e9bcc"}
Oct 31 16:00:01.629003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.628935    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" podStartSLOduration=1.62891422 podStartE2EDuration="1.62891422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.626708921 +0000 UTC m=+4442.352989901" watchObservedRunningTime="2026-10-31 16:00:01.62891422 +0000 UTC m=+4442.355195198"
Oct 31 16:00:01.742224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.742158    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" podStartSLOduration=1.742144653 podStartE2EDuration="1.742144653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.738040991 +0000 UTC m=+4442.464321969" watchObservedRunningTime="2026-10-31 16:00:01.742144653 +0000 UTC m=+4442.468425628"
Oct 31 16:00:01.851386 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:01.851327    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" podStartSLOduration=1.851311412 podStartE2EDuration="1.851311412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.849095467 +0000 UTC m=+4442.575376446" watchObservedRunningTime="2026-10-31 16:00:01.851311412 +0000 UTC m=+4442.577592389"
Oct 31 16:00:02.532597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.532549    4549 generic.go:356] "Generic (PLEG): container finished" podID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerID="60731aa312ec8a0647ba0415fa6a27b098bd4bad2dde065fa02ad78a928dbf5b" exitCode=60
Oct 31 16:00:02.533095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.532641    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerDied","Data":"60731aa312ec8a0647ba0415fa6a27b098bd4bad2dde065fa02ad78a928dbf5b"}
Oct 31 16:00:02.533095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.532936    4549 scope.go:122] "RemoveContainer" containerID="60731aa312ec8a0647ba0415fa6a27b098bd4bad2dde065fa02ad78a928dbf5b"
Oct 31 16:00:02.535742 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.535716    4549 generic.go:356] "Generic (PLEG): container finished" podID="5313f442-4afc-4c29-a87e-a0a45573263e" containerID="03ba45fc59f4fe78daceca3be97adc9061ad6781160e9c1793e42289b9ad06bb" exitCode=60
Oct 31 16:00:02.535929 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.535834    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerDied","Data":"03ba45fc59f4fe78daceca3be97adc9061ad6781160e9c1793e42289b9ad06bb"}
Oct 31 16:00:02.536159 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.536137    4549 scope.go:122] "RemoveContainer" containerID="03ba45fc59f4fe78daceca3be97adc9061ad6781160e9c1793e42289b9ad06bb"
Oct 31 16:00:02.538129 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.538111    4549 generic.go:356] "Generic (PLEG): container finished" podID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerID="b0c1abbaf65649e3a600796f1c065355bc584f036b92c60aed23fecb1da6baed" exitCode=60
Oct 31 16:00:02.538237 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.538196    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerDied","Data":"b0c1abbaf65649e3a600796f1c065355bc584f036b92c60aed23fecb1da6baed"}
Oct 31 16:00:02.538428 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:02.538413    4549 scope.go:122] "RemoveContainer" containerID="b0c1abbaf65649e3a600796f1c065355bc584f036b92c60aed23fecb1da6baed"
Oct 31 16:00:03.560112 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:03.560073    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerStarted","Data":"28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"}
Oct 31 16:00:03.562273 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:03.562245    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerStarted","Data":"4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"}
Oct 31 16:00:03.564232 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:03.564211    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerStarted","Data":"a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"}
Oct 31 16:00:04.572441 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.572408    4549 generic.go:356] "Generic (PLEG): container finished" podID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerID="28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb" exitCode=60
Oct 31 16:00:04.572956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.572499    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerDied","Data":"28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"}
Oct 31 16:00:04.572956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.572563    4549 scope.go:122] "RemoveContainer" containerID="60731aa312ec8a0647ba0415fa6a27b098bd4bad2dde065fa02ad78a928dbf5b"
Oct 31 16:00:04.572956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.572802    4549 scope.go:122] "RemoveContainer" containerID="28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"
Oct 31 16:00:04.573133 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:04.573114    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84_odh-ai-gateway-infra(f768c934-6c16-487e-92e8-2ce06b6ab105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105"
Oct 31 16:00:04.574556 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.574537    4549 generic.go:356] "Generic (PLEG): container finished" podID="5313f442-4afc-4c29-a87e-a0a45573263e" containerID="4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d" exitCode=60
Oct 31 16:00:04.574688 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.574600    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerDied","Data":"4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"}
Oct 31 16:00:04.574883 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.574868    4549 scope.go:122] "RemoveContainer" containerID="4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"
Oct 31 16:00:04.575147 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:04.575128    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf_odh-ai-gateway-infra(5313f442-4afc-4c29-a87e-a0a45573263e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" podUID="5313f442-4afc-4c29-a87e-a0a45573263e"
Oct 31 16:00:04.576510 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.576490    4549 generic.go:356] "Generic (PLEG): container finished" podID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerID="a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99" exitCode=60
Oct 31 16:00:04.576627 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.576590    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerDied","Data":"a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"}
Oct 31 16:00:04.576800 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.576786    4549 scope.go:122] "RemoveContainer" containerID="a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"
Oct 31 16:00:04.577045 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:04.577029    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs_odh-ai-gateway-infra(613ce2e0-e194-42f3-bc31-e9ba7fad4cda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda"
Oct 31 16:00:04.587061 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.587036    4549 scope.go:122] "RemoveContainer" containerID="03ba45fc59f4fe78daceca3be97adc9061ad6781160e9c1793e42289b9ad06bb"
Oct 31 16:00:04.603235 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:04.603213    4549 scope.go:122] "RemoveContainer" containerID="b0c1abbaf65649e3a600796f1c065355bc584f036b92c60aed23fecb1da6baed"
Oct 31 16:00:05.585381 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:05.585348    4549 scope.go:122] "RemoveContainer" containerID="28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"
Oct 31 16:00:05.585909 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:05.585689    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84_odh-ai-gateway-infra(f768c934-6c16-487e-92e8-2ce06b6ab105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105"
Oct 31 16:00:05.587108 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:05.587091    4549 scope.go:122] "RemoveContainer" containerID="4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"
Oct 31 16:00:05.587374 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:05.587351    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf_odh-ai-gateway-infra(5313f442-4afc-4c29-a87e-a0a45573263e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" podUID="5313f442-4afc-4c29-a87e-a0a45573263e"
Oct 31 16:00:05.588875 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:05.588855    4549 scope.go:122] "RemoveContainer" containerID="a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"
Oct 31 16:00:05.589109 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:05.589087    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs_odh-ai-gateway-infra(613ce2e0-e194-42f3-bc31-e9ba7fad4cda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda"
Oct 31 16:00:07.019445 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:07.019411    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:00:14.412444 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:14.412411    4549 scope.go:122] "RemoveContainer" containerID="4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"
Oct 31 16:00:15.412986 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:15.412916    4549 scope.go:122] "RemoveContainer" containerID="28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"
Oct 31 16:00:15.678854 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:15.678767    4549 generic.go:356] "Generic (PLEG): container finished" podID="5313f442-4afc-4c29-a87e-a0a45573263e" containerID="44918b81b3765bf9c4cde247248eb53c1f0acc750b01740bdaaf00d8b757a857" exitCode=60
Oct 31 16:00:15.678854 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:15.678834    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerDied","Data":"44918b81b3765bf9c4cde247248eb53c1f0acc750b01740bdaaf00d8b757a857"}
Oct 31 16:00:15.679100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:15.678912    4549 scope.go:122] "RemoveContainer" containerID="4e4b9c2969833f39407b7fce60b84811877cb7221dcc86b31efd7df3ea8d932d"
Oct 31 16:00:15.680227 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:15.679337    4549 scope.go:122] "RemoveContainer" containerID="44918b81b3765bf9c4cde247248eb53c1f0acc750b01740bdaaf00d8b757a857"
Oct 31 16:00:15.680773 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:15.680735    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf_odh-ai-gateway-infra(5313f442-4afc-4c29-a87e-a0a45573263e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" podUID="5313f442-4afc-4c29-a87e-a0a45573263e"
Oct 31 16:00:16.686678 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.686643    4549 generic.go:356] "Generic (PLEG): container finished" podID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerID="f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513" exitCode=0
Oct 31 16:00:16.687199 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.686716    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerDied","Data":"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"}
Oct 31 16:00:16.687199 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.687154    4549 scope.go:122] "RemoveContainer" containerID="f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"
Oct 31 16:00:16.688821 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688801    4549 generic.go:356] "Generic (PLEG): container finished" podID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerID="7e3518db7d3706ba50e2a2267e540d2ba5bc1017d43524cb749e807c626afc97" exitCode=60
Oct 31 16:00:16.688821 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688799    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:00:16.688963 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688878    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerDied","Data":"7e3518db7d3706ba50e2a2267e540d2ba5bc1017d43524cb749e807c626afc97"}
Oct 31 16:00:16.688963 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688884    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:00:16.688963 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688925    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:00:16.688963 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.688935    4549 scope.go:122] "RemoveContainer" containerID="28708ea3f1c19dce147e3850cbc3bb1654133b3fb2e8bfc0dd2b2b2a011dd1bb"
Oct 31 16:00:16.689284 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.689264    4549 scope.go:122] "RemoveContainer" containerID="7e3518db7d3706ba50e2a2267e540d2ba5bc1017d43524cb749e807c626afc97"
Oct 31 16:00:16.689583 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:16.689537    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84_odh-ai-gateway-infra(f768c934-6c16-487e-92e8-2ce06b6ab105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105"
Oct 31 16:00:16.804533 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.804499    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:16.965960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:16.965926    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:17.122627 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.122561    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\") pod \"5313f442-4afc-4c29-a87e-a0a45573263e\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") "
Oct 31 16:00:17.122811 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.122729    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\" (UniqueName: \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\") pod \"5313f442-4afc-4c29-a87e-a0a45573263e\" (UID: \"5313f442-4afc-4c29-a87e-a0a45573263e\") "
Oct 31 16:00:17.122942 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.122916    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca" pod "5313f442-4afc-4c29-a87e-a0a45573263e" (UID: "5313f442-4afc-4c29-a87e-a0a45573263e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:00:17.123260 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.123244    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5313f442-4afc-4c29-a87e-a0a45573263e-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:17.125294 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.125265    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp" pod "5313f442-4afc-4c29-a87e-a0a45573263e" (UID: "5313f442-4afc-4c29-a87e-a0a45573263e"). InnerVolumeSpecName "kube-api-access-dq5wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:17.224493 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.224420    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq5wp\" (UniqueName: \"kubernetes.io/projected/5313f442-4afc-4c29-a87e-a0a45573263e-kube-api-access-dq5wp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:17.415595 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.414140    4549 scope.go:122] "RemoveContainer" containerID="a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"
Oct 31 16:00:17.723796 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.720550    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerStarted","Data":"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"}
Oct 31 16:00:17.726802 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.726770    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf" event={"ID":"5313f442-4afc-4c29-a87e-a0a45573263e","Type":"ContainerDied","Data":"ecf6be348f866b5d51b00fde609f6d89dc3d3158286a6e40f8cc52f44cdb3478"}
Oct 31 16:00:17.726802 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.726792    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"
Oct 31 16:00:17.727026 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.726844    4549 scope.go:122] "RemoveContainer" containerID="44918b81b3765bf9c4cde247248eb53c1f0acc750b01740bdaaf00d8b757a857"
Oct 31 16:00:17.731635 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.731603    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:00:17.732996 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.732970    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:17.733113 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.733019    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:17.952638 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.952531    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:17.955865 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.955839    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-55b3fb-29891040-dbdqf"]
Oct 31 16:00:17.986362 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:17.986326    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:18.186317 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.186295    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:18.338908 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.338872    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\" (UniqueName: \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\") pod \"f768c934-6c16-487e-92e8-2ce06b6ab105\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") "
Oct 31 16:00:18.339119 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.338983    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\") pod \"f768c934-6c16-487e-92e8-2ce06b6ab105\" (UID: \"f768c934-6c16-487e-92e8-2ce06b6ab105\") "
Oct 31 16:00:18.339367 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.339342    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca" pod "f768c934-6c16-487e-92e8-2ce06b6ab105" (UID: "f768c934-6c16-487e-92e8-2ce06b6ab105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:00:18.341148 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.341125    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp" pod "f768c934-6c16-487e-92e8-2ce06b6ab105" (UID: "f768c934-6c16-487e-92e8-2ce06b6ab105"). InnerVolumeSpecName "kube-api-access-rl2fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:18.440481 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.440449    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f768c934-6c16-487e-92e8-2ce06b6ab105-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:18.440481 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.440476    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2fp\" (UniqueName: \"kubernetes.io/projected/f768c934-6c16-487e-92e8-2ce06b6ab105-kube-api-access-rl2fp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:18.730886 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.730798    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:18.730886 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.730853    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:18.738629 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.738595    4549 generic.go:356] "Generic (PLEG): container finished" podID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerID="1afda2cac612632eecb86f4861432579318a91d2cbbe87fff7388413b762be56" exitCode=60
Oct 31 16:00:18.738801 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.738678    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerDied","Data":"1afda2cac612632eecb86f4861432579318a91d2cbbe87fff7388413b762be56"}
Oct 31 16:00:18.738801 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.738739    4549 scope.go:122] "RemoveContainer" containerID="a349cf533c6ef99faad33f60c02fb5e1bc1cc1349ac6f0d92861c8e43ee1dd99"
Oct 31 16:00:18.739394 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.739378    4549 scope.go:122] "RemoveContainer" containerID="1afda2cac612632eecb86f4861432579318a91d2cbbe87fff7388413b762be56"
Oct 31 16:00:18.739861 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:18.739831    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs_odh-ai-gateway-infra(613ce2e0-e194-42f3-bc31-e9ba7fad4cda)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda"
Oct 31 16:00:18.741697 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.741666    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84" event={"ID":"f768c934-6c16-487e-92e8-2ce06b6ab105","Type":"ContainerDied","Data":"b437f9ea960b795a7d269747a706eab83be1ed9693b232c25e2f47358c030014"}
Oct 31 16:00:18.741812 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.741684    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"
Oct 31 16:00:18.752368 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.752351    4549 scope.go:122] "RemoveContainer" containerID="7e3518db7d3706ba50e2a2267e540d2ba5bc1017d43524cb749e807c626afc97"
Oct 31 16:00:18.978634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.978600    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:18.981819 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:18.981755    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891040-mcd84"]
Oct 31 16:00:19.442455 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:19.442424    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" path="/var/lib/kubelet/pods/5313f442-4afc-4c29-a87e-a0a45573263e/volumes"
Oct 31 16:00:19.442880 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:19.442866    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" path="/var/lib/kubelet/pods/f768c934-6c16-487e-92e8-2ce06b6ab105/volumes"
Oct 31 16:00:19.730989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:19.730900    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:19.730989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:19.730966    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:19.861284 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:19.861244    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:20.050133 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.050101    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:20.160449 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.160412    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\") pod \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") "
Oct 31 16:00:20.160671 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.160451    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\" (UniqueName: \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\") pod \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\" (UID: \"613ce2e0-e194-42f3-bc31-e9ba7fad4cda\") "
Oct 31 16:00:20.160804 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.160781    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca" pod "613ce2e0-e194-42f3-bc31-e9ba7fad4cda" (UID: "613ce2e0-e194-42f3-bc31-e9ba7fad4cda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:00:20.162841 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.162819    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw" pod "613ce2e0-e194-42f3-bc31-e9ba7fad4cda" (UID: "613ce2e0-e194-42f3-bc31-e9ba7fad4cda"). InnerVolumeSpecName "kube-api-access-vddfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:20.261967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.261932    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:20.261967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.261962    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddfw\" (UniqueName: \"kubernetes.io/projected/613ce2e0-e194-42f3-bc31-e9ba7fad4cda-kube-api-access-vddfw\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:20.731396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.731357    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:20.731831 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.731410    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:20.758310 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.758283    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"
Oct 31 16:00:20.758486 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.758280    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs" event={"ID":"613ce2e0-e194-42f3-bc31-e9ba7fad4cda","Type":"ContainerDied","Data":"ae05c6e63fbe61c8449017e3c22b4a8eaabecd06db8d4527d2e9985d3c6e9bcc"}
Oct 31 16:00:20.758486 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.758420    4549 scope.go:122] "RemoveContainer" containerID="1afda2cac612632eecb86f4861432579318a91d2cbbe87fff7388413b762be56"
Oct 31 16:00:20.930728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.930687    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:20.933608 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:20.933588    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-190693-29891040-7kjrs"]
Oct 31 16:00:21.433160 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:21.433131    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" path="/var/lib/kubelet/pods/613ce2e0-e194-42f3-bc31-e9ba7fad4cda/volumes"
Oct 31 16:00:21.731151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:21.731066    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:21.731151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:21.731121    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:22.020590 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:22.017732    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:00:22.731501 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:22.731463    4549 patch_prober.go:29] interesting pod/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 31 16:00:22.731738 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:22.731518    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 31 16:00:23.737035 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:23.737006    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:00:23.738331 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:23.738310    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:00:27.762136 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:27.762104    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 16:00:27.762554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:27.762351    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" podUID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" containerName="maas-api" containerID="cri-o://a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee" gracePeriod=30
Oct 31 16:00:28.077327 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.077303    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 16:00:28.161704 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.161660    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\") pod \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") "
Oct 31 16:00:28.162442 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.162412    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\") pod \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") "
Oct 31 16:00:28.162634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.162481    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\") pod \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") "
Oct 31 16:00:28.162634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.162525    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") pod \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\" (UID: \"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8\") "
Oct 31 16:00:28.163258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.163225    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca" pod "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" (UID: "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:00:28.164966 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.164947    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:28.165488 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.165463    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt" pod "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" (UID: "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8"). InnerVolumeSpecName "kube-api-access-ztrtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:28.167659 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.167635    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs" pod "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" (UID: "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:00:28.167794 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.167777    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls" pod "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" (UID: "c85fa0a0-4dcd-4e8b-9cf0-8653794958d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:00:28.266206 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.266172    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrtt\" (UniqueName: \"kubernetes.io/projected/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-kube-api-access-ztrtt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:28.266206 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.266200    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:28.266206 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.266216    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:28.813674 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.813633    4549 generic.go:356] "Generic (PLEG): container finished" podID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" containerID="a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee" exitCode=0
Oct 31 16:00:28.814185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.813697    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"
Oct 31 16:00:28.814185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.813703    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" event={"ID":"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8","Type":"ContainerDied","Data":"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"}
Oct 31 16:00:28.814185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.813757    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx" event={"ID":"c85fa0a0-4dcd-4e8b-9cf0-8653794958d8","Type":"ContainerDied","Data":"87f07c63772307ade991fc9893b8db61141952fc85e211a515b22e5c92d92e8b"}
Oct 31 16:00:28.814185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.813790    4549 scope.go:122] "RemoveContainer" containerID="a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"
Oct 31 16:00:28.826340 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.826317    4549 scope.go:122] "RemoveContainer" containerID="a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"
Oct 31 16:00:28.826650 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:28.826623    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee\": container with ID starting with a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee not found: ID does not exist" containerID="a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"
Oct 31 16:00:28.826741 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.826659    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee"} err="failed to get container status \"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee\": rpc error: code = NotFound desc = could not find container \"a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee\": container with ID starting with a6dc74dfcc772e19f1eed6c17a10cf8235a8868c831f17aafbb8c5cca78068ee not found: ID does not exist"
Oct 31 16:00:28.976930 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.976881    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 16:00:28.979926 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:28.979901    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c947d9-855f7fdf7b-kr4dx"]
Oct 31 16:00:29.421517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:29.421481    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" path="/var/lib/kubelet/pods/c85fa0a0-4dcd-4e8b-9cf0-8653794958d8/volumes"
Oct 31 16:00:37.012270 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:37.012229    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:00:38.341441 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:38.341405    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 16:00:38.390280 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:38.390249    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 16:00:42.312550 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.312512    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:00:42.313270 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313251    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.313324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313274    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.313324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313304    4549 cpu_manager.go:414] "containerMap: removing container" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.313324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313314    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313328    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" containerName="maas-api"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313337    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" containerName="maas-api"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313354    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313361    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313372    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313379    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313393    4549 cpu_manager.go:414] "containerMap: removing container" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313400    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313414    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313421    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313445    4549 cpu_manager.go:414] "containerMap: removing container" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.313487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313452    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313593    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313606    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313616    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313628    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313640    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313649    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85fa0a0-4dcd-4e8b-9cf0-8653794958d8" containerName="maas-api"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313656    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313663    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:00:42.314072 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.313675    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="613ce2e0-e194-42f3-bc31-e9ba7fad4cda" containerName="cleanup"
Oct 31 16:00:42.317445 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.317425    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.370370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.370327    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"openshift-service-ca.crt\""
Oct 31 16:00:42.370616 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.370448    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"rate-test-model-aa864d-kserve-self-signed-certs\""
Oct 31 16:00:42.370616 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.370486    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"kube-root-ca.crt\""
Oct 31 16:00:42.370616 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.370530    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-aa864d\"/\"default-dockercfg-6m78b\""
Oct 31 16:00:42.371999 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.371964    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:00:42.431054 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431024    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.431275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431073    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.431275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431123    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsb8\" (UniqueName: \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.431275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431144    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.431275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431226    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.431275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.431248    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.532640 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532605    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.532842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532668    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.532842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532713    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.532842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532765    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.532842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532830    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsb8\" (UniqueName: \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.533095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.532860    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.533475 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.533211    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.533475 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.533369    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.533844 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.533781    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.535291 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.535272    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.536356 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.536334    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.590456 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.590380    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsb8\" (UniqueName: \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\") pod \"rate-test-model-aa864d-kserve-65c676fbd5-h8hcr\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") " pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:42.656590 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:42.656535    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:43.040246 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:43.040215    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:00:43.042416 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:00:43.042389    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07ee4042_2df2_4e64_a04f_6ce60bb1a2ca.slice/crio-a7ea6aadc2d1751f2f797f449bd64179a0542946d74a93a886e829802da40e2a WatchSource:0}: Error finding container a7ea6aadc2d1751f2f797f449bd64179a0542946d74a93a886e829802da40e2a: Status 404 returned error can't find the container with id a7ea6aadc2d1751f2f797f449bd64179a0542946d74a93a886e829802da40e2a
Oct 31 16:00:43.918710 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:43.918675    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" event={"ID":"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca","Type":"ContainerStarted","Data":"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"}
Oct 31 16:00:43.918710 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:43.918717    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" event={"ID":"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca","Type":"ContainerStarted","Data":"a7ea6aadc2d1751f2f797f449bd64179a0542946d74a93a886e829802da40e2a"}
Oct 31 16:00:44.034975 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:44.034908    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" podStartSLOduration=2.034889831 podStartE2EDuration="2.034889831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:44.031245829 +0000 UTC m=+4484.757526993" watchObservedRunningTime="2026-10-31 16:00:44.034889831 +0000 UTC m=+4484.761170810"
Oct 31 16:00:44.657371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:44.657336    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:52.656976 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:52.656936    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:52.670107 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:52.670083    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:52.992799 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:52.992720    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:00:53.364466 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.364409    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main" containerID="cri-o://e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb" gracePeriod=45
Oct 31 16:00:53.413767 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.413730    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main" containerID="cri-o://86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92" gracePeriod=45
Oct 31 16:00:53.684774 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.684727    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 16:00:53.713859 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.713832    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 16:00:53.776944 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.776908    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\" (UniqueName: \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.776967    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777007    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777095 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777039    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777160    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\" (UniqueName: \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777209    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777249    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\" (UniqueName: \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777260    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777265    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.777293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777290    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777321    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777359    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777388    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\") pod \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\" (UID: \"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70\") "
Oct 31 16:00:53.777680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777453    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\") pod \"a1240204-96ae-4de2-a443-af97313eec1b\" (UID: \"a1240204-96ae-4de2-a443-af97313eec1b\") "
Oct 31 16:00:53.777905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777852    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.777905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777868    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.778017 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.777904    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.778086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.778072    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.778145 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.778084    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.778145 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.778129    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.780185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.780156    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "kube-api-access-x689l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:53.780565 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.780539    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "kube-api-access-wbqfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:00:53.780695 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.780625    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:00:53.780695 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.780642    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:00:53.780858 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.780841    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm" pod "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" (UID: "bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.781100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.781079    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm" pod "a1240204-96ae-4de2-a443-af97313eec1b" (UID: "a1240204-96ae-4de2-a443-af97313eec1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:00:53.879481 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879451    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879481 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879479    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1240204-96ae-4de2-a443-af97313eec1b-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879489    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbqfr\" (UniqueName: \"kubernetes.io/projected/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-kube-api-access-wbqfr\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879499    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879507    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879514    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879522    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879530    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879542    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x689l\" (UniqueName: \"kubernetes.io/projected/a1240204-96ae-4de2-a443-af97313eec1b-kube-api-access-x689l\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.879728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.879553    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1240204-96ae-4de2-a443-af97313eec1b-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:00:53.990735 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.990650    4549 generic.go:356] "Generic (PLEG): container finished" podID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerID="e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb" exitCode=0
Oct 31 16:00:53.990735 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.990720    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"
Oct 31 16:00:53.990949 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.990729    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" event={"ID":"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70","Type":"ContainerDied","Data":"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"}
Oct 31 16:00:53.990949 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.990775    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p" event={"ID":"bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70","Type":"ContainerDied","Data":"fc38287829180cdaeb6d1eb07e702721ec2395c9725b9321eeb111969e2f5f79"}
Oct 31 16:00:53.990949 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.990798    4549 scope.go:122] "RemoveContainer" containerID="e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"
Oct 31 16:00:53.992447 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.992423    4549 generic.go:356] "Generic (PLEG): container finished" podID="a1240204-96ae-4de2-a443-af97313eec1b" containerID="86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92" exitCode=0
Oct 31 16:00:53.992757 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.992650    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" event={"ID":"a1240204-96ae-4de2-a443-af97313eec1b","Type":"ContainerDied","Data":"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"}
Oct 31 16:00:53.992757 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.992696    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn" event={"ID":"a1240204-96ae-4de2-a443-af97313eec1b","Type":"ContainerDied","Data":"db28e1935ce841737783fedc2769190ccb61ede85e5343001e0afbd3366f9fb7"}
Oct 31 16:00:53.992882 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:53.992804    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"
Oct 31 16:00:54.001419 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.001399    4549 scope.go:122] "RemoveContainer" containerID="e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"
Oct 31 16:00:54.001993 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:54.001754    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb\": container with ID starting with e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb not found: ID does not exist" containerID="e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"
Oct 31 16:00:54.001993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.001789    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb"} err="failed to get container status \"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb\": rpc error: code = NotFound desc = could not find container \"e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb\": container with ID starting with e9b2c342383374cdaeb65e932df16e332f7fe2480fd53a19cff6f0906a44a3bb not found: ID does not exist"
Oct 31 16:00:54.001993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.001811    4549 scope.go:122] "RemoveContainer" containerID="86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"
Oct 31 16:00:54.011383 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.011360    4549 scope.go:122] "RemoveContainer" containerID="86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"
Oct 31 16:00:54.011677 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:00:54.011657    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92\": container with ID starting with 86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92 not found: ID does not exist" containerID="86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"
Oct 31 16:00:54.011737 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.011685    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92"} err="failed to get container status \"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92\": rpc error: code = NotFound desc = could not find container \"86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92\": container with ID starting with 86ea7b8bdef7c01cebcffb90c003d307681309e1006f0344595f71cec481cc92 not found: ID does not exist"
Oct 31 16:00:54.166189 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.166156    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 16:00:54.179496 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.179468    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-w2-029d26-kserve-54c5df7dd8-qbkbn"]
Oct 31 16:00:54.298124 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.298094    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 16:00:54.300796 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:54.300775    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-029d26/distinct-2-w2-029d26-kserve-76bb6db7f-hsx7p"]
Oct 31 16:00:55.419965 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:55.419930    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1240204-96ae-4de2-a443-af97313eec1b" path="/var/lib/kubelet/pods/a1240204-96ae-4de2-a443-af97313eec1b/volumes"
Oct 31 16:00:55.420398 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:00:55.420347    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" path="/var/lib/kubelet/pods/bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70/volumes"
Oct 31 16:01:03.199266 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:03.199239    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:01:03.352490 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:03.352455    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:01:07.011942 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:01:07.011906    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:01:29.155382 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155349    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155846    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155860    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155881    4549 cpu_manager.go:414] "containerMap: removing container" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155887    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155902    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155906    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155919    4549 cpu_manager.go:414] "containerMap: removing container" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:01:29.155967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155924    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5313f442-4afc-4c29-a87e-a0a45573263e" containerName="cleanup"
Oct 31 16:01:29.156230 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.155992    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec4cd1e-46c9-45d0-aa8e-49ab69ab6a70" containerName="main"
Oct 31 16:01:29.156230 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.156001    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="f768c934-6c16-487e-92e8-2ce06b6ab105" containerName="cleanup"
Oct 31 16:01:29.156230 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.156008    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1240204-96ae-4de2-a443-af97313eec1b" containerName="main"
Oct 31 16:01:29.158431 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.158416    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.210358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.210323    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-029d26-openshift-default-dockercfg-lxm75\""
Oct 31 16:01:29.263387 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.263353    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:01:29.277760 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.277732    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.277932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.277773    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.277932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.277809    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.277932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.277900    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.278073 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.277952    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.278073 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.278045    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsjf\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.278154 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.278072    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.278154 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.278088    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.278154 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.278141    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.379807 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379772    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379820    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379847    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379882    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379908    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsjf\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379946    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.379970    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380351 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380171    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380351 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380286    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380351 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380321    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380351 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380325    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380532 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380408    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380603 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380552    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.380683 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.380666    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.382303 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.382283    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.382820 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.382804    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.437543 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.437469    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.437763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.437741    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsjf\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.471453 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.471422    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:29.903522 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.903490    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:01:29.905360 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:01:29.905325    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef99d2bb_89af_4ecf_ba17_f2df483115de.slice/crio-6164fca2f891094f6b3c196d63431c8635c0e3c8dbd3ac060d3c5423d979800d WatchSource:0}: Error finding container 6164fca2f891094f6b3c196d63431c8635c0e3c8dbd3ac060d3c5423d979800d: Status 404 returned error can't find the container with id 6164fca2f891094f6b3c196d63431c8635c0e3c8dbd3ac060d3c5423d979800d
Oct 31 16:01:29.908327 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.908288    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:01:29.908420 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.908373    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:01:29.908467 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:29.908416    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:01:30.227938 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.227841    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" event={"ID":"ef99d2bb-89af-4ecf-ba17-f2df483115de","Type":"ContainerStarted","Data":"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"}
Oct 31 16:01:30.227938 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.227899    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" event={"ID":"ef99d2bb-89af-4ecf-ba17-f2df483115de","Type":"ContainerStarted","Data":"6164fca2f891094f6b3c196d63431c8635c0e3c8dbd3ac060d3c5423d979800d"}
Oct 31 16:01:30.350418 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.350350    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podStartSLOduration=1.350328586 podStartE2EDuration="1.350328586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:30.346921468 +0000 UTC m=+4531.073202459" watchObservedRunningTime="2026-10-31 16:01:30.350328586 +0000 UTC m=+4531.076609568"
Oct 31 16:01:30.472598 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.472547    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:30.474133 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.474102    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:30.474272 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:30.474154    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:31.472394 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:31.472352    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:31.472913 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:31.472409    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:32.472002 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:32.471964    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:32.472203 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:32.472029    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:33.471869 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:33.471837    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:33.472294 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:33.471886    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:34.471864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:34.471830    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:34.472076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:34.471883    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:35.472349 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:35.472310    4549 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 31 16:01:35.472776 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:35.472380    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 31 16:01:36.495240 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:36.495208    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:36.495783 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:36.495705    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:36.497812 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:36.496652    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:37.024636 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:01:37.024561    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:01:50.221362 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.221280    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 16:01:50.221840 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.221483    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" podUID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" containerName="maas-api" containerID="cri-o://54f32d506712b1f248f04a1922c0794c52bdc29da7972b05472dfe54e53464e7" gracePeriod=30
Oct 31 16:01:50.368687 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.368655    4549 generic.go:356] "Generic (PLEG): container finished" podID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" containerID="54f32d506712b1f248f04a1922c0794c52bdc29da7972b05472dfe54e53464e7" exitCode=0
Oct 31 16:01:50.368874 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.368730    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" event={"ID":"1aa8b25f-3105-4fbd-aea0-060e3de136fd","Type":"ContainerDied","Data":"54f32d506712b1f248f04a1922c0794c52bdc29da7972b05472dfe54e53464e7"}
Oct 31 16:01:50.530772 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.530745    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 16:01:50.642145 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642109    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\") pod \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") "
Oct 31 16:01:50.642325 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642180    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\" (UniqueName: \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\") pod \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") "
Oct 31 16:01:50.642325 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642233    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\") pod \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") "
Oct 31 16:01:50.642325 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642263    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") pod \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\" (UID: \"1aa8b25f-3105-4fbd-aea0-060e3de136fd\") "
Oct 31 16:01:50.642643 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642615    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca" pod "1aa8b25f-3105-4fbd-aea0-060e3de136fd" (UID: "1aa8b25f-3105-4fbd-aea0-060e3de136fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:01:50.642806 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.642783    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa8b25f-3105-4fbd-aea0-060e3de136fd-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:50.645172 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.645143    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs" pod "1aa8b25f-3105-4fbd-aea0-060e3de136fd" (UID: "1aa8b25f-3105-4fbd-aea0-060e3de136fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:01:50.645296 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.645204    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp" pod "1aa8b25f-3105-4fbd-aea0-060e3de136fd" (UID: "1aa8b25f-3105-4fbd-aea0-060e3de136fd"). InnerVolumeSpecName "kube-api-access-rjlmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:01:50.645457 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.645432    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls" pod "1aa8b25f-3105-4fbd-aea0-060e3de136fd" (UID: "1aa8b25f-3105-4fbd-aea0-060e3de136fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:01:50.743880 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.743846    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:50.743880 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.743875    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlmp\" (UniqueName: \"kubernetes.io/projected/1aa8b25f-3105-4fbd-aea0-060e3de136fd-kube-api-access-rjlmp\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:50.743880 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:50.743889    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa8b25f-3105-4fbd-aea0-060e3de136fd-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:51.377075 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:51.377043    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"
Oct 31 16:01:51.377075 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:51.377062    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl" event={"ID":"1aa8b25f-3105-4fbd-aea0-060e3de136fd","Type":"ContainerDied","Data":"c73912a333668cdc692bb2317ca6fe1f4086466be92944dbcc587bb7e6b2fc92"}
Oct 31 16:01:51.377653 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:51.377122    4549 scope.go:122] "RemoveContainer" containerID="54f32d506712b1f248f04a1922c0794c52bdc29da7972b05472dfe54e53464e7"
Oct 31 16:01:51.540112 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:51.540077    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 16:01:51.589845 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:51.589810    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-029d26-9c758649-px7nl"]
Oct 31 16:01:53.418955 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:53.418923    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" path="/var/lib/kubelet/pods/1aa8b25f-3105-4fbd-aea0-060e3de136fd/volumes"
Oct 31 16:01:54.629212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:54.629177    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:01:54.629764 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:54.629396    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy" containerID="cri-o://0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923" gracePeriod=30
Oct 31 16:01:59.887018 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.886989    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:01:59.954952 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.954862    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.954952 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.954911    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.954994    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955043    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955082    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955117    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955154    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955241    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955280    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\") pod \"ef99d2bb-89af-4ecf-ba17-f2df483115de\" (UID: \"ef99d2bb-89af-4ecf-ba17-f2df483115de\") "
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955309    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955327    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955359    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:01:59.955461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955432    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:01:59.955801 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955712    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:01:59.955863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955826    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef99d2bb-89af-4ecf-ba17-f2df483115de-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:59.955863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955848    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:59.955863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955860    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:59.955863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955869    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:59.955863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.955878    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:01:59.957531 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.957501    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:01:59.957689 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.957666    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "kube-api-access-zcsjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:01:59.957979 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.957961    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:01:59.958049 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:01:59.957976    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy" pod "ef99d2bb-89af-4ecf-ba17-f2df483115de" (UID: "ef99d2bb-89af-4ecf-ba17-f2df483115de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:02:00.057076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.057045    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:02:00.057076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.057075    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:02:00.057311 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.057091    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcsjf\" (UniqueName: \"kubernetes.io/projected/ef99d2bb-89af-4ecf-ba17-f2df483115de-kube-api-access-zcsjf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:02:00.057311 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.057106    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef99d2bb-89af-4ecf-ba17-f2df483115de-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:02:00.441956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.441919    4549 generic.go:356] "Generic (PLEG): container finished" podID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerID="0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923" exitCode=0
Oct 31 16:02:00.442597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.441976    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"
Oct 31 16:02:00.442597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.442007    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" event={"ID":"ef99d2bb-89af-4ecf-ba17-f2df483115de","Type":"ContainerDied","Data":"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"}
Oct 31 16:02:00.442597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.442077    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j" event={"ID":"ef99d2bb-89af-4ecf-ba17-f2df483115de","Type":"ContainerDied","Data":"6164fca2f891094f6b3c196d63431c8635c0e3c8dbd3ac060d3c5423d979800d"}
Oct 31 16:02:00.442597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.442110    4549 scope.go:122] "RemoveContainer" containerID="0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"
Oct 31 16:02:00.453877 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.453854    4549 scope.go:122] "RemoveContainer" containerID="0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"
Oct 31 16:02:00.454337 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:02:00.454306    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923\": container with ID starting with 0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923 not found: ID does not exist" containerID="0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"
Oct 31 16:02:00.454453 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.454345    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923"} err="failed to get container status \"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923\": rpc error: code = NotFound desc = could not find container \"0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923\": container with ID starting with 0eec09aa03fd53886a7174a04288c4be3b4a26e4195ded8bf63b838f19dea923 not found: ID does not exist"
Oct 31 16:02:00.606734 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.606697    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:02:00.654509 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:00.654473    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-tnv7j"]
Oct 31 16:02:01.421099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:01.421062    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" path="/var/lib/kubelet/pods/ef99d2bb-89af-4ecf-ba17-f2df483115de/volumes"
Oct 31 16:02:07.012319 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:02:07.012279    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:02:37.024156 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:02:37.024111    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:02:44.646883 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.646842    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:02:44.647491 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647360    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" containerName="maas-api"
Oct 31 16:02:44.647491 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647374    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" containerName="maas-api"
Oct 31 16:02:44.647491 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647395    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy"
Oct 31 16:02:44.647491 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647415    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy"
Oct 31 16:02:44.647491 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647486    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa8b25f-3105-4fbd-aea0-060e3de136fd" containerName="maas-api"
Oct 31 16:02:44.647825 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.647498    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef99d2bb-89af-4ecf-ba17-f2df483115de" containerName="istio-proxy"
Oct 31 16:02:44.650985 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.650960    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.707485 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.707454    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:02:44.754035 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.754003    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-140c48\"/\"kube-root-ca.crt\""
Oct 31 16:02:44.754247 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.754120    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-140c48\"/\"openshift-service-ca.crt\""
Oct 31 16:02:44.754247 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.754214    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-140c48\"/\"default-dockercfg-wn7sd\""
Oct 31 16:02:44.756303 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.756275    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-140c48\"/\"sub-test-model-140c48-kserve-self-signed-certs\""
Oct 31 16:02:44.762894 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.762868    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtzpg\" (UniqueName: \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.763032 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.762971    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.763032 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.763002    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.763141 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.763114    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.763190 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.763176    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.763278 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.763267    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864560 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864524    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864618    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864662    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864698    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtzpg\" (UniqueName: \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864735    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.864778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.864754    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.865157 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.865132    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.865403 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.865377    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.865709 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.865684    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.867931 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.867903    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.868051 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.868009    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.925033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.924956    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtzpg\" (UniqueName: \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\") pod \"sub-test-model-140c48-kserve-5f4c5ddff7-x5hml\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") " pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:44.964560 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:44.964517    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:45.375286 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:02:45.375250    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6735b72_4477_4929_84ba_a271819780e4.slice/crio-1aea564c59468380c9c294e41b24b81bb9f17cd244a03e4a4b39e9c7fdad0b33 WatchSource:0}: Error finding container 1aea564c59468380c9c294e41b24b81bb9f17cd244a03e4a4b39e9c7fdad0b33: Status 404 returned error can't find the container with id 1aea564c59468380c9c294e41b24b81bb9f17cd244a03e4a4b39e9c7fdad0b33
Oct 31 16:02:45.379337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:45.379306    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:02:45.729992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:45.729899    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" event={"ID":"a6735b72-4477-4929-84ba-a271819780e4","Type":"ContainerStarted","Data":"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"}
Oct 31 16:02:45.729992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:45.729945    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" event={"ID":"a6735b72-4477-4929-84ba-a271819780e4","Type":"ContainerStarted","Data":"1aea564c59468380c9c294e41b24b81bb9f17cd244a03e4a4b39e9c7fdad0b33"}
Oct 31 16:02:45.844234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:45.844168    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" podStartSLOduration=1.844148159 podStartE2EDuration="1.844148159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:02:45.842552658 +0000 UTC m=+4606.568833638" watchObservedRunningTime="2026-10-31 16:02:45.844148159 +0000 UTC m=+4606.570429138"
Oct 31 16:02:45.964973 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:45.964926    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:54.965770 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:54.965734    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:54.978715 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:54.978692    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:02:55.805363 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:02:55.805337    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:03:07.011265 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:03:07.011222    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:03:37.012929 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:03:37.012847    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:04:07.013026 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:07.012981    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:04:20.406802 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:20.406761    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 16:04:20.407353 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:20.407038    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" containerID="cri-o://d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad" gracePeriod=30
Oct 31 16:04:23.156135 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:23.156097    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 16:04:25.188232 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.188196    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-dnbd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:04:25.188692 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.188269    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:04:25.685322 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.685297    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 16:04:25.829911 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.829879    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830091 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.829932    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830091 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.829976    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830091 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830062    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830240 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830091    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830240 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830175    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830356 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830237    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830356 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830302    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830356 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830278    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:25.830356 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830343    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\") pod \"71b63b5b-d342-499b-80d3-f9d241157934\" (UID: \"71b63b5b-d342-499b-80d3-f9d241157934\") "
Oct 31 16:04:25.830629 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830373    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:25.830629 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830561    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:25.830824 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830762    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:25.830916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830860    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.830985 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.830918    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.831224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.831039    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.831224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.831063    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:25.832612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.832554    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:04:25.832733 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.832673    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:25.832733 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.832675    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:25.832958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.832939    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk" pod "71b63b5b-d342-499b-80d3-f9d241157934" (UID: "71b63b5b-d342-499b-80d3-f9d241157934"). InnerVolumeSpecName "kube-api-access-dk8vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:25.931902 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931866    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.931902 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931898    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71b63b5b-d342-499b-80d3-f9d241157934-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.931902 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931908    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8vk\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-kube-api-access-dk8vk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.932156 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931920    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71b63b5b-d342-499b-80d3-f9d241157934-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.932156 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931929    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71b63b5b-d342-499b-80d3-f9d241157934-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:25.932156 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:25.931937    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71b63b5b-d342-499b-80d3-f9d241157934-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:26.391777 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.391747    4549 generic.go:356] "Generic (PLEG): container finished" podID="71b63b5b-d342-499b-80d3-f9d241157934" containerID="d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad" exitCode=0
Oct 31 16:04:26.392204 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.391807    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"
Oct 31 16:04:26.392204 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.391835    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" event={"ID":"71b63b5b-d342-499b-80d3-f9d241157934","Type":"ContainerDied","Data":"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"}
Oct 31 16:04:26.392204 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.391921    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5" event={"ID":"71b63b5b-d342-499b-80d3-f9d241157934","Type":"ContainerDied","Data":"2aaf6c2a5824f7cc67d22fa8e1c0cfa0653363a9849dcdfe0b3b47d806db2216"}
Oct 31 16:04:26.392204 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.391948    4549 scope.go:122] "RemoveContainer" containerID="d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"
Oct 31 16:04:26.403030 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.403010    4549 scope.go:122] "RemoveContainer" containerID="d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"
Oct 31 16:04:26.403313 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:26.403295    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad\": container with ID starting with d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad not found: ID does not exist" containerID="d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"
Oct 31 16:04:26.403380 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.403328    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad"} err="failed to get container status \"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad\": rpc error: code = NotFound desc = could not find container \"d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad\": container with ID starting with d8cd79e0110773acaf80070f8d4e0b9744fd972f5bbf3880374f953a678a2dad not found: ID does not exist"
Oct 31 16:04:26.558397 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.558356    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 16:04:26.558947 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:26.558924    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-dnbd5"]
Oct 31 16:04:27.418231 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:27.418197    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b63b5b-d342-499b-80d3-f9d241157934" path="/var/lib/kubelet/pods/71b63b5b-d342-499b-80d3-f9d241157934/volumes"
Oct 31 16:04:37.012563 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:37.012521    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:04:38.176627 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.176542    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerName="main" containerID="cri-o://6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273" gracePeriod=45
Oct 31 16:04:38.433972 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.433892    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 16:04:38.474542 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.474506    4549 generic.go:356] "Generic (PLEG): container finished" podID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerID="6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273" exitCode=0
Oct 31 16:04:38.474747 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.474552    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"
Oct 31 16:04:38.474747 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.474593    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" event={"ID":"3bf45538-caf4-4eca-8401-206d4b655e2f","Type":"ContainerDied","Data":"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"}
Oct 31 16:04:38.474747 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.474644    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" event={"ID":"3bf45538-caf4-4eca-8401-206d4b655e2f","Type":"ContainerDied","Data":"97a863ba7f645ffc68630342e0e77c25dee8c5e82ed6c1271dc7cd760dd9cc47"}
Oct 31 16:04:38.474747 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.474667    4549 scope.go:122] "RemoveContainer" containerID="6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"
Oct 31 16:04:38.484897 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.484876    4549 scope.go:122] "RemoveContainer" containerID="6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"
Oct 31 16:04:38.485154 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:38.485133    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273\": container with ID starting with 6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273 not found: ID does not exist" containerID="6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"
Oct 31 16:04:38.485211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.485164    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273"} err="failed to get container status \"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273\": rpc error: code = NotFound desc = could not find container \"6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273\": container with ID starting with 6b13ea547e2c40d37fdb3c1faee82269153fb395694a5a1396afce7d4b428273 not found: ID does not exist"
Oct 31 16:04:38.569324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569296    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.569517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569360    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.569517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569401    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.569517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569418    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.569517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569478    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\" (UniqueName: \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.569795 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569697    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:38.569795 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569713    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:38.569920 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.569798    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\") pod \"3bf45538-caf4-4eca-8401-206d4b655e2f\" (UID: \"3bf45538-caf4-4eca-8401-206d4b655e2f\") "
Oct 31 16:04:38.570055 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.570027    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:38.570434 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.570415    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.570508 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.570440    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.570508 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.570455    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.571978 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.571947    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:38.572088 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.571993    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "kube-api-access-7txrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:38.572088 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.572076    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs" pod "3bf45538-caf4-4eca-8401-206d4b655e2f" (UID: "3bf45538-caf4-4eca-8401-206d4b655e2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:38.671506 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.671468    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf45538-caf4-4eca-8401-206d4b655e2f-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.671506 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.671503    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf45538-caf4-4eca-8401-206d4b655e2f-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.671506 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.671514    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txrb\" (UniqueName: \"kubernetes.io/projected/3bf45538-caf4-4eca-8401-206d4b655e2f-kube-api-access-7txrb\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:38.895813 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.895771    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 16:04:38.901488 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:38.901458    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v"]
Oct 31 16:04:39.368849 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:39.368797    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-e72453/rate-test-model-e72453-kserve-587cdf4f7c-wdr7v" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.184:8000/ready\": context deadline exceeded"
Oct 31 16:04:39.419056 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:39.419023    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" path="/var/lib/kubelet/pods/3bf45538-caf4-4eca-8401-206d4b655e2f/volumes"
Oct 31 16:04:43.346027 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:43.345945    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:04:50.636503 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:50.636465    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 16:04:50.636974 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:50.636702    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" containerID="cri-o://e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3" gracePeriod=30
Oct 31 16:04:52.333079 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:52.333039    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mc5xl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:04:52.333529 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:52.333143    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:04:55.907352 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.907319    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 16:04:55.972458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972381    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972437    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972455    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972521    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972557    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972611    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972639    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972712    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.972756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972742    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\") pod \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\" (UID: \"85568dbe-008f-4a0a-b567-a5e2a072d1dc\") "
Oct 31 16:04:55.973111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.972847    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:55.973111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973047    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:55.973111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973042    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:04:55.973286 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973247    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:55.973286 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973254    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:55.973286 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973267    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:55.973647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973285    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:55.973647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.973290    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:55.974963 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.974939    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:04:55.975084 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.975007    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:55.975084 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.975023    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "kube-api-access-ppssk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:55.975250 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:55.975223    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy" pod "85568dbe-008f-4a0a-b567-a5e2a072d1dc" (UID: "85568dbe-008f-4a0a-b567-a5e2a072d1dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:56.074889 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074852    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.074889 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074889    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.074889 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074902    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.075215 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074914    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.075215 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074926    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppssk\" (UniqueName: \"kubernetes.io/projected/85568dbe-008f-4a0a-b567-a5e2a072d1dc-kube-api-access-ppssk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.075215 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.074938    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85568dbe-008f-4a0a-b567-a5e2a072d1dc-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:56.631698 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.631662    4549 generic.go:356] "Generic (PLEG): container finished" podID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerID="e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3" exitCode=0
Oct 31 16:04:56.632112 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.631731    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"
Oct 31 16:04:56.632248 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.631750    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" event={"ID":"85568dbe-008f-4a0a-b567-a5e2a072d1dc","Type":"ContainerDied","Data":"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"}
Oct 31 16:04:56.632314 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.632250    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl" event={"ID":"85568dbe-008f-4a0a-b567-a5e2a072d1dc","Type":"ContainerDied","Data":"c5830dc0058dbda9b2c39425610d7ddb72e38e198fa048f17ab6542cbcf62601"}
Oct 31 16:04:56.632314 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.632282    4549 scope.go:122] "RemoveContainer" containerID="e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"
Oct 31 16:04:56.643980 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.643954    4549 scope.go:122] "RemoveContainer" containerID="e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"
Oct 31 16:04:56.644270 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:56.644252    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3\": container with ID starting with e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3 not found: ID does not exist" containerID="e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"
Oct 31 16:04:56.644354 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.644279    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3"} err="failed to get container status \"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3\": rpc error: code = NotFound desc = could not find container \"e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3\": container with ID starting with e38d110a9b29855c866bab89ce07b1780f1f622979ce570f2c04f39af9d9dca3 not found: ID does not exist"
Oct 31 16:04:56.845700 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.845663    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 16:04:56.895328 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:56.895250    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mc5xl"]
Oct 31 16:04:57.420628 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:57.420594    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" path="/var/lib/kubelet/pods/85568dbe-008f-4a0a-b567-a5e2a072d1dc/volumes"
Oct 31 16:04:58.366826 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.366790    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" podUID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" containerName="main" containerID="cri-o://e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417" gracePeriod=45
Oct 31 16:04:58.641013 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.640986    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:04:58.647252 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.647223    4549 generic.go:356] "Generic (PLEG): container finished" podID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" containerID="e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417" exitCode=0
Oct 31 16:04:58.647411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.647277    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"
Oct 31 16:04:58.647411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.647312    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" event={"ID":"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca","Type":"ContainerDied","Data":"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"}
Oct 31 16:04:58.647411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.647367    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr" event={"ID":"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca","Type":"ContainerDied","Data":"a7ea6aadc2d1751f2f797f449bd64179a0542946d74a93a886e829802da40e2a"}
Oct 31 16:04:58.647411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.647388    4549 scope.go:122] "RemoveContainer" containerID="e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"
Oct 31 16:04:58.658448 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.658431    4549 scope.go:122] "RemoveContainer" containerID="e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"
Oct 31 16:04:58.658733 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:04:58.658713    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417\": container with ID starting with e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417 not found: ID does not exist" containerID="e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"
Oct 31 16:04:58.658819 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.658741    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417"} err="failed to get container status \"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417\": rpc error: code = NotFound desc = could not find container \"e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417\": container with ID starting with e05db247dc0955d629627f02132d6f45eb5ec7a606b7f9bbfd8eb10362ea2417 not found: ID does not exist"
Oct 31 16:04:58.701810 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.701774    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702017 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.701831    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\" (UniqueName: \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702017 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.701865    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702017 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.701918    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702047    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702074    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\") pod \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\" (UID: \"07ee4042-2df2-4e64-a04f-6ce60bb1a2ca\") "
Oct 31 16:04:58.702337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702241    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:58.702337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702307    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:58.702471 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702449    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:58.702961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.702940    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:58.703162 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.703139    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:58.703162 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.703162    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:58.704677 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.704514    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:04:58.704952 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.704931    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "kube-api-access-xxsb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:04:58.705142 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.705118    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs" pod "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" (UID: "07ee4042-2df2-4e64-a04f-6ce60bb1a2ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:04:58.804546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.804500    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsb8\" (UniqueName: \"kubernetes.io/projected/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-kube-api-access-xxsb8\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:58.804546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.804545    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:58.804546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:58.804558    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:04:59.068689 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:59.068652    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:04:59.071188 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:59.071168    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-aa864d/rate-test-model-aa864d-kserve-65c676fbd5-h8hcr"]
Oct 31 16:04:59.421924 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:04:59.421887    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" path="/var/lib/kubelet/pods/07ee4042-2df2-4e64-a04f-6ce60bb1a2ca/volumes"
Oct 31 16:05:00.940440 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:00.940406    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 16:05:07.014822 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:05:07.014779    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:05:15.962022 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:15.961970    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main" containerID="cri-o://9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d" gracePeriod=45
Oct 31 16:05:16.228158 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.227734    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 16:05:16.295265 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295233    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295296    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295336    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\" (UniqueName: \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295402    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295432    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295740 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295593    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\" (UniqueName: \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\") pod \"2de67f02-20c2-4163-b8b2-961280068c46\" (UID: \"2de67f02-20c2-4163-b8b2-961280068c46\") "
Oct 31 16:05:16.295740 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295594    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:16.295740 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295664    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:16.296009 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.295992    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.296064 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.296015    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.296294 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.296247    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:16.298523 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.298500    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:05:16.298965 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.298944    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "kube-api-access-m5qwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:05:16.299663 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.299646    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm" pod "2de67f02-20c2-4163-b8b2-961280068c46" (UID: "2de67f02-20c2-4163-b8b2-961280068c46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:16.396750 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.396712    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.396750 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.396752    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2de67f02-20c2-4163-b8b2-961280068c46-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.396978 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.396766    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2de67f02-20c2-4163-b8b2-961280068c46-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.396978 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.396781    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qwx\" (UniqueName: \"kubernetes.io/projected/2de67f02-20c2-4163-b8b2-961280068c46-kube-api-access-m5qwx\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:16.775732 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.775695    4549 generic.go:356] "Generic (PLEG): container finished" podID="2de67f02-20c2-4163-b8b2-961280068c46" containerID="9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d" exitCode=0
Oct 31 16:05:16.775943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.775748    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"
Oct 31 16:05:16.775943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.775774    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" event={"ID":"2de67f02-20c2-4163-b8b2-961280068c46","Type":"ContainerDied","Data":"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"}
Oct 31 16:05:16.775943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.775813    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh" event={"ID":"2de67f02-20c2-4163-b8b2-961280068c46","Type":"ContainerDied","Data":"9c6f798c321605eebcca651b6420edcecef3d93afa370604369e52c218c4732a"}
Oct 31 16:05:16.775943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.775829    4549 scope.go:122] "RemoveContainer" containerID="9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"
Oct 31 16:05:16.787496 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.787477    4549 scope.go:122] "RemoveContainer" containerID="9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"
Oct 31 16:05:16.787839 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:05:16.787818    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d\": container with ID starting with 9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d not found: ID does not exist" containerID="9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"
Oct 31 16:05:16.787913 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.787854    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d"} err="failed to get container status \"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d\": rpc error: code = NotFound desc = could not find container \"9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d\": container with ID starting with 9b4c2177d5025cb608471dd4123aff50fb9443b4cf113880895cf4a026ecc16d not found: ID does not exist"
Oct 31 16:05:16.940395 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.940362    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 16:05:16.988137 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:16.988107    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-2-w3-190693-kserve-848c56b577-fgswh"]
Oct 31 16:05:17.418768 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:17.418739    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de67f02-20c2-4163-b8b2-961280068c46" path="/var/lib/kubelet/pods/2de67f02-20c2-4163-b8b2-961280068c46/volumes"
Oct 31 16:05:23.346185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346134    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346587    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerName="main"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346600    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerName="main"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346612    4549 cpu_manager.go:414] "containerMap: removing container" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346617    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346628    4549 cpu_manager.go:414] "containerMap: removing container" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346633    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main"
Oct 31 16:05:23.346651 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346654    4549 cpu_manager.go:414] "containerMap: removing container" podUID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" containerName="main"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346660    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" containerName="main"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346678    4549 cpu_manager.go:414] "containerMap: removing container" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346683    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346740    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ee4042-2df2-4e64-a04f-6ce60bb1a2ca" containerName="main"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346751    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b63b5b-d342-499b-80d3-f9d241157934" containerName="istio-proxy"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346756    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf45538-caf4-4eca-8401-206d4b655e2f" containerName="main"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346761    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="85568dbe-008f-4a0a-b567-a5e2a072d1dc" containerName="istio-proxy"
Oct 31 16:05:23.346916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.346767    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de67f02-20c2-4163-b8b2-961280068c46" containerName="main"
Oct 31 16:05:23.352258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.352221    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.404292 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.404258    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\""
Oct 31 16:05:23.456801 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.456770    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:23.464654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.464602    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.464842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.464705    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.464842 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.464795    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.464984 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.464842    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.465235 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.465182    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6gf\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.465421 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.465360    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.465421 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.465397    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.465596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.465531    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.465673 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.465612    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566093 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566053    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566093 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566093    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566210    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6gf\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566292    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566324    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566447    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566476    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566507    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566538    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566845 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566820    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.566845 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566836    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.567042 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566928    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.567042 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566943    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.567042 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.566943    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.568831 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.568809    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.568913 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.568898    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.622944 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.622862    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6gf\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.622944 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.622877    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-dj2qt\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:23.669324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:23.669291    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:24.097494 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.097469    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:24.098877 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:05:24.098849    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7107cbd7_f4e5_47b8_842e_95c89d8d952e.slice/crio-1662c1767b7688cffcf2767587549bbf8467ec1e819d3d03550369cd4e0da442 WatchSource:0}: Error finding container 1662c1767b7688cffcf2767587549bbf8467ec1e819d3d03550369cd4e0da442: Status 404 returned error can't find the container with id 1662c1767b7688cffcf2767587549bbf8467ec1e819d3d03550369cd4e0da442
Oct 31 16:05:24.101455 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.101416    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:05:24.101545 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.101503    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:05:24.101618 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.101546    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:05:24.833906 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.833874    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" event={"ID":"7107cbd7-f4e5-47b8-842e-95c89d8d952e","Type":"ContainerStarted","Data":"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"}
Oct 31 16:05:24.834438 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:24.833928    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" event={"ID":"7107cbd7-f4e5-47b8-842e-95c89d8d952e","Type":"ContainerStarted","Data":"1662c1767b7688cffcf2767587549bbf8467ec1e819d3d03550369cd4e0da442"}
Oct 31 16:05:25.010398 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:25.010316    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podStartSLOduration=2.010296649 podStartE2EDuration="2.010296649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:24.997681392 +0000 UTC m=+4765.723962375" watchObservedRunningTime="2026-10-31 16:05:25.010296649 +0000 UTC m=+4765.736577622"
Oct 31 16:05:25.670007 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:25.669969    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:25.671359 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:25.671333    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:25.671501 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:25.671401    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:26.670510 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:26.670474    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:26.671049 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:26.670527    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:27.670332 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:27.670298    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:27.670523 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:27.670346    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:28.669718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:28.669681    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:28.670135 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:28.669739    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:29.670254 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:29.670210    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:29.670687 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:29.670285    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:30.669829 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:30.669791    4549 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 31 16:05:30.670018 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:30.669859    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 31 16:05:31.822021 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:31.821984    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:31.822698 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:31.822669    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:31.823690 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:31.823664    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:37.013271 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:05:37.013225    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:05:51.352722 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:51.352688    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:51.353376 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:51.352984    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy" containerID="cri-o://822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5" gracePeriod=30
Oct 31 16:05:56.617698 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.617666    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:56.753970 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.753881    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.753970 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.753930    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.753970 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.753963    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754004    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754036    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754143    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754169    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754207    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754223    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754286    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754327    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:56.754396 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754312    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\") pod \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\" (UID: \"7107cbd7-f4e5-47b8-842e-95c89d8d952e\") "
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754509    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754520    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754828    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754849    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754865    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754879    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.754900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.754894    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.756545 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.756515    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:05:56.756691 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.756623    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:05:56.757050 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.757032    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:05:56.757157 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.757140    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf" pod "7107cbd7-f4e5-47b8-842e-95c89d8d952e" (UID: "7107cbd7-f4e5-47b8-842e-95c89d8d952e"). InnerVolumeSpecName "kube-api-access-qz6gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:05:56.855788 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.855752    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.855788 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.855779    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.855788 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.855789    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7107cbd7-f4e5-47b8-842e-95c89d8d952e-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:56.855788 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:56.855799    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6gf\" (UniqueName: \"kubernetes.io/projected/7107cbd7-f4e5-47b8-842e-95c89d8d952e-kube-api-access-qz6gf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:05:57.038245 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.038210    4549 generic.go:356] "Generic (PLEG): container finished" podID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerID="822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5" exitCode=0
Oct 31 16:05:57.038434 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.038265    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"
Oct 31 16:05:57.038434 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.038303    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" event={"ID":"7107cbd7-f4e5-47b8-842e-95c89d8d952e","Type":"ContainerDied","Data":"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"}
Oct 31 16:05:57.038434 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.038372    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt" event={"ID":"7107cbd7-f4e5-47b8-842e-95c89d8d952e","Type":"ContainerDied","Data":"1662c1767b7688cffcf2767587549bbf8467ec1e819d3d03550369cd4e0da442"}
Oct 31 16:05:57.038434 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.038403    4549 scope.go:122] "RemoveContainer" containerID="822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"
Oct 31 16:05:57.050433 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.050413    4549 scope.go:122] "RemoveContainer" containerID="822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"
Oct 31 16:05:57.050737 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:05:57.050715    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5\": container with ID starting with 822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5 not found: ID does not exist" containerID="822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"
Oct 31 16:05:57.050801 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.050746    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5"} err="failed to get container status \"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5\": rpc error: code = NotFound desc = could not find container \"822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5\": container with ID starting with 822882a8c692165b9a6eb912521494a8c69d411875ebe3f00d78e2dc69fa49c5 not found: ID does not exist"
Oct 31 16:05:57.202374 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.202345    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:57.252061 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.252026    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-dj2qt"]
Oct 31 16:05:57.420291 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:05:57.420207    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" path="/var/lib/kubelet/pods/7107cbd7-f4e5-47b8-842e-95c89d8d952e/volumes"
Oct 31 16:06:03.243330 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:03.243301    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:06:03.399711 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:03.399667    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:06:07.011461 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:06:07.011428    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:06:14.634022 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:14.633942    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 16:06:29.655245 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.655199    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main" containerID="cri-o://e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6" gracePeriod=45
Oct 31 16:06:29.908536 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.908476    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 16:06:29.998681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998646    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.998884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998707    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\" (UniqueName: \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.998884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998755    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.998884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998791    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.998884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998833    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\" (UniqueName: \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.999115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998936    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\") pod \"1859f7ae-117c-435f-8cf4-83d1d456e128\" (UID: \"1859f7ae-117c-435f-8cf4-83d1d456e128\") "
Oct 31 16:06:29.999115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998985    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:06:29.999115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.998998    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:06:29.999261 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.999235    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:06:29.999548 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.999440    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:29.999645 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.999592    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:29.999645 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:29.999615    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:30.001099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.001069    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "kube-api-access-l847w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:06:30.001099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.001076    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:06:30.001099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.001084    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm" pod "1859f7ae-117c-435f-8cf4-83d1d456e128" (UID: "1859f7ae-117c-435f-8cf4-83d1d456e128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:06:30.100739 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.100702    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l847w\" (UniqueName: \"kubernetes.io/projected/1859f7ae-117c-435f-8cf4-83d1d456e128-kube-api-access-l847w\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:30.100739 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.100741    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1859f7ae-117c-435f-8cf4-83d1d456e128-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:30.101007 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.100755    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1859f7ae-117c-435f-8cf4-83d1d456e128-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:30.247982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.247896    4549 generic.go:356] "Generic (PLEG): container finished" podID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerID="e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6" exitCode=0
Oct 31 16:06:30.247982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.247968    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"
Oct 31 16:06:30.248185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.247985    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" event={"ID":"1859f7ae-117c-435f-8cf4-83d1d456e128","Type":"ContainerDied","Data":"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"}
Oct 31 16:06:30.248185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.248028    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd" event={"ID":"1859f7ae-117c-435f-8cf4-83d1d456e128","Type":"ContainerDied","Data":"ee4818070ff7545851e6ffce1afc1e45a5ecb696c26d008b5f0f217ad386d532"}
Oct 31 16:06:30.248185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.248046    4549 scope.go:122] "RemoveContainer" containerID="e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"
Oct 31 16:06:30.260654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.260423    4549 scope.go:122] "RemoveContainer" containerID="e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"
Oct 31 16:06:30.260924 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:06:30.260901    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6\": container with ID starting with e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6 not found: ID does not exist" containerID="e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"
Oct 31 16:06:30.261031 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.260937    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6"} err="failed to get container status \"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6\": rpc error: code = NotFound desc = could not find container \"e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6\": container with ID starting with e413449300dbc8bc630eda2a36633a099569f3256e548a7e55ea897d739c1fb6 not found: ID does not exist"
Oct 31 16:06:30.414584 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.414527    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 16:06:30.461175 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:30.461137    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2348/distinct-w1-fb2348-kserve-6f4f4f88f5-2qrkd"]
Oct 31 16:06:31.418319 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:31.418275    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" path="/var/lib/kubelet/pods/1859f7ae-117c-435f-8cf4-83d1d456e128/volumes"
Oct 31 16:06:37.011271 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:06:37.011231    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:06:42.571268 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.571227    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 16:06:42.571795 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.571425    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" podUID="92463f88-bab2-4936-9c20-350c03b92071" containerName="maas-api" containerID="cri-o://ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c" gracePeriod=30
Oct 31 16:06:42.845719 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.845689    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 16:06:42.934033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.933995    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\" (UniqueName: \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\") pod \"92463f88-bab2-4936-9c20-350c03b92071\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") "
Oct 31 16:06:42.934255 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.934110    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\") pod \"92463f88-bab2-4936-9c20-350c03b92071\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") "
Oct 31 16:06:42.934255 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.934178    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\") pod \"92463f88-bab2-4936-9c20-350c03b92071\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") "
Oct 31 16:06:42.934255 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.934220    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\") pod \"92463f88-bab2-4936-9c20-350c03b92071\" (UID: \"92463f88-bab2-4936-9c20-350c03b92071\") "
Oct 31 16:06:42.934588 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.934534    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca" pod "92463f88-bab2-4936-9c20-350c03b92071" (UID: "92463f88-bab2-4936-9c20-350c03b92071"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:06:42.936554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.936507    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42" pod "92463f88-bab2-4936-9c20-350c03b92071" (UID: "92463f88-bab2-4936-9c20-350c03b92071"). InnerVolumeSpecName "kube-api-access-wmz42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:06:42.936554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.936545    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs" pod "92463f88-bab2-4936-9c20-350c03b92071" (UID: "92463f88-bab2-4936-9c20-350c03b92071"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:06:42.936782 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:42.936765    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls" pod "92463f88-bab2-4936-9c20-350c03b92071" (UID: "92463f88-bab2-4936-9c20-350c03b92071"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:06:43.035986 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.035953    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92463f88-bab2-4936-9c20-350c03b92071-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:43.035986 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.035985    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:43.035986 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.035996    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmz42\" (UniqueName: \"kubernetes.io/projected/92463f88-bab2-4936-9c20-350c03b92071-kube-api-access-wmz42\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:43.036255 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.036013    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92463f88-bab2-4936-9c20-350c03b92071-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:06:43.324919 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.324880    4549 generic.go:356] "Generic (PLEG): container finished" podID="92463f88-bab2-4936-9c20-350c03b92071" containerID="ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c" exitCode=0
Oct 31 16:06:43.325111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.324924    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"
Oct 31 16:06:43.325111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.324974    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" event={"ID":"92463f88-bab2-4936-9c20-350c03b92071","Type":"ContainerDied","Data":"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"}
Oct 31 16:06:43.325111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.325038    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn" event={"ID":"92463f88-bab2-4936-9c20-350c03b92071","Type":"ContainerDied","Data":"f90242350d2ed6beb2a46265a404d5c42334c5657211c3e7449579e265c897a9"}
Oct 31 16:06:43.325111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.325069    4549 scope.go:122] "RemoveContainer" containerID="ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"
Oct 31 16:06:43.337558 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.337535    4549 scope.go:122] "RemoveContainer" containerID="ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"
Oct 31 16:06:43.337866 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:06:43.337841    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c\": container with ID starting with ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c not found: ID does not exist" containerID="ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"
Oct 31 16:06:43.337945 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.337881    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c"} err="failed to get container status \"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c\": rpc error: code = NotFound desc = could not find container \"ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c\": container with ID starting with ee8d14ce27d8cf52e8bf019a9713301f6d514c008663c4dfda7dc63f17e5be8c not found: ID does not exist"
Oct 31 16:06:43.497160 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.497122    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 16:06:43.497343 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:43.497186    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-fb2348-6f48bf766b-64xwn"]
Oct 31 16:06:45.418966 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:06:45.418932    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92463f88-bab2-4936-9c20-350c03b92071" path="/var/lib/kubelet/pods/92463f88-bab2-4936-9c20-350c03b92071/volumes"
Oct 31 16:07:07.012629 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:07:07.012590    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:07:37.012696 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:07:37.012650    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:08:07.012875 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:08:07.012837    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:08:34.249013 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:34.248982    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:08:37.012162 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:08:37.012127    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:08:42.920646 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:42.920600    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 16:08:43.027672 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:43.027639    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 16:08:44.319034 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.318984    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 16:08:44.319555 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.319279    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" podUID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" containerName="maas-api" containerID="cri-o://00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065" gracePeriod=30
Oct 31 16:08:44.606841 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.606809    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 16:08:44.677283 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.677247    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") pod \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") "
Oct 31 16:08:44.677283 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.677286    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\") pod \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") "
Oct 31 16:08:44.677556 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.677338    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\") pod \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") "
Oct 31 16:08:44.677556 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.677415    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\" (UniqueName: \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\") pod \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\" (UID: \"8b9dae8a-d3e1-49f4-b540-5055b58aefd5\") "
Oct 31 16:08:44.677734 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.677706    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca" pod "8b9dae8a-d3e1-49f4-b540-5055b58aefd5" (UID: "8b9dae8a-d3e1-49f4-b540-5055b58aefd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:08:44.679683 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.679646    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls" pod "8b9dae8a-d3e1-49f4-b540-5055b58aefd5" (UID: "8b9dae8a-d3e1-49f4-b540-5055b58aefd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:44.679807 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.679719    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7" pod "8b9dae8a-d3e1-49f4-b540-5055b58aefd5" (UID: "8b9dae8a-d3e1-49f4-b540-5055b58aefd5"). InnerVolumeSpecName "kube-api-access-nwtb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:08:44.679807 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.679715    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs" pod "8b9dae8a-d3e1-49f4-b540-5055b58aefd5" (UID: "8b9dae8a-d3e1-49f4-b540-5055b58aefd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:44.778521 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.778481    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtb7\" (UniqueName: \"kubernetes.io/projected/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-kube-api-access-nwtb7\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:44.778521 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.778517    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:44.778773 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.778532    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:44.778773 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:44.778545    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b9dae8a-d3e1-49f4-b540-5055b58aefd5-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:45.070233 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.070194    4549 generic.go:356] "Generic (PLEG): container finished" podID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" containerID="00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065" exitCode=0
Oct 31 16:08:45.070463 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.070239    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"
Oct 31 16:08:45.070463 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.070283    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" event={"ID":"8b9dae8a-d3e1-49f4-b540-5055b58aefd5","Type":"ContainerDied","Data":"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"}
Oct 31 16:08:45.070463 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.070342    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5" event={"ID":"8b9dae8a-d3e1-49f4-b540-5055b58aefd5","Type":"ContainerDied","Data":"f1d7458f2920913a95d9ca0a3582382b25768a8f9344660725fb98bc17925b46"}
Oct 31 16:08:45.070463 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.070372    4549 scope.go:122] "RemoveContainer" containerID="00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"
Oct 31 16:08:45.081392 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.081365    4549 scope.go:122] "RemoveContainer" containerID="00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"
Oct 31 16:08:45.081669 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:08:45.081645    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065\": container with ID starting with 00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065 not found: ID does not exist" containerID="00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"
Oct 31 16:08:45.081777 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.081681    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065"} err="failed to get container status \"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065\": rpc error: code = NotFound desc = could not find container \"00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065\": container with ID starting with 00148321c1732dda91cfcaa543fce89248607cdc94a94f4bbdc87e744556f065 not found: ID does not exist"
Oct 31 16:08:45.233375 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.233340    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 16:08:45.239171 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.239146    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-31dff8-56f6869bcc-mnxw5"]
Oct 31 16:08:45.419141 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:45.419063    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" path="/var/lib/kubelet/pods/8b9dae8a-d3e1-49f4-b540-5055b58aefd5/volumes"
Oct 31 16:08:47.921123 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:47.921084    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing" containerID="cri-o://a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a" gracePeriod=25
Oct 31 16:08:48.028333 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:48.028255    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing" containerID="cri-o://c35b42a95a74622867ecf86f5e904f9e9f65d5b83bddcc01b9cc26f1d45d2dd3" gracePeriod=25
Oct 31 16:08:49.268981 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.268935    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" podUID="a6735b72-4477-4929-84ba-a271819780e4" containerName="main" containerID="cri-o://b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378" gracePeriod=45
Oct 31 16:08:49.533134 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.533078    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:08:49.624516 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624485    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624529    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624649    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\" (UniqueName: \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624672    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624708    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624751    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\") pod \"a6735b72-4477-4929-84ba-a271819780e4\" (UID: \"a6735b72-4477-4929-84ba-a271819780e4\") "
Oct 31 16:08:49.624960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624817    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:08:49.624960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.624913    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:08:49.625147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.625094    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:49.625147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.625113    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:49.625147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.625091    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:08:49.627191 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.627170    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:08:49.627318 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.627258    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:08:49.627366 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.627334    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg" pod "a6735b72-4477-4929-84ba-a271819780e4" (UID: "a6735b72-4477-4929-84ba-a271819780e4"). InnerVolumeSpecName "kube-api-access-mtzpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:08:49.726679 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.726643    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtzpg\" (UniqueName: \"kubernetes.io/projected/a6735b72-4477-4929-84ba-a271819780e4-kube-api-access-mtzpg\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:49.726679 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.726674    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6735b72-4477-4929-84ba-a271819780e4-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:49.726679 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.726684    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:49.726923 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:49.726693    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6735b72-4477-4929-84ba-a271819780e4-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:08:50.100980 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.100944    4549 generic.go:356] "Generic (PLEG): container finished" podID="a6735b72-4477-4929-84ba-a271819780e4" containerID="b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378" exitCode=0
Oct 31 16:08:50.101165 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.101000    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"
Oct 31 16:08:50.101165 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.101036    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" event={"ID":"a6735b72-4477-4929-84ba-a271819780e4","Type":"ContainerDied","Data":"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"}
Oct 31 16:08:50.101165 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.101087    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml" event={"ID":"a6735b72-4477-4929-84ba-a271819780e4","Type":"ContainerDied","Data":"1aea564c59468380c9c294e41b24b81bb9f17cd244a03e4a4b39e9c7fdad0b33"}
Oct 31 16:08:50.101165 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.101109    4549 scope.go:122] "RemoveContainer" containerID="b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"
Oct 31 16:08:50.111994 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.111977    4549 scope.go:122] "RemoveContainer" containerID="b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"
Oct 31 16:08:50.112298 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:08:50.112277    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378\": container with ID starting with b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378 not found: ID does not exist" containerID="b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"
Oct 31 16:08:50.112371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.112311    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378"} err="failed to get container status \"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378\": rpc error: code = NotFound desc = could not find container \"b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378\": container with ID starting with b5cd62cb5a027e247c6466cba20a861ed4d2363c5c313e69e4ac6adaf7f7c378 not found: ID does not exist"
Oct 31 16:08:50.265916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.265882    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:08:50.315552 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:50.315518    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-140c48/sub-test-model-140c48-kserve-5f4c5ddff7-x5hml"]
Oct 31 16:08:51.420300 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:51.420260    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6735b72-4477-4929-84ba-a271819780e4" path="/var/lib/kubelet/pods/a6735b72-4477-4929-84ba-a271819780e4/volumes"
Oct 31 16:08:56.925110 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:56.925064    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:08:56.925110 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:08:56.925086    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:09:06.925155 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:06.925117    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:09:06.925668 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:06.925125    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:09:07.011159 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:07.011122    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:09:08.181172 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.181109    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 16:09:08.213528 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.213496    4549 generic.go:356] "Generic (PLEG): container finished" podID="9a398640-2695-466c-a576-8d241f12838f" containerID="c35b42a95a74622867ecf86f5e904f9e9f65d5b83bddcc01b9cc26f1d45d2dd3" exitCode=0
Oct 31 16:09:08.213693 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.213552    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" event={"ID":"9a398640-2695-466c-a576-8d241f12838f","Type":"ContainerDied","Data":"c35b42a95a74622867ecf86f5e904f9e9f65d5b83bddcc01b9cc26f1d45d2dd3"}
Oct 31 16:09:08.215025 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.215003    4549 generic.go:356] "Generic (PLEG): container finished" podID="c7b09986-6611-4d04-9706-2b4c291c9720" containerID="a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a" exitCode=0
Oct 31 16:09:08.215168 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.215060    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"
Oct 31 16:09:08.215168 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.215061    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" event={"ID":"c7b09986-6611-4d04-9706-2b4c291c9720","Type":"ContainerDied","Data":"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"}
Oct 31 16:09:08.215267 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.215199    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd" event={"ID":"c7b09986-6611-4d04-9706-2b4c291c9720","Type":"ContainerDied","Data":"30ceea3bd539ba0ff27c3abbc6a6dd082c295f0a90465ee0d71be8f18a80d956"}
Oct 31 16:09:08.215267 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.215240    4549 scope.go:122] "RemoveContainer" containerID="a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"
Oct 31 16:09:08.226235 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.226208    4549 scope.go:122] "RemoveContainer" containerID="a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"
Oct 31 16:09:08.226642 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:08.226620    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a\": container with ID starting with a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a not found: ID does not exist" containerID="a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"
Oct 31 16:09:08.226751 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.226655    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a"} err="failed to get container status \"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a\": rpc error: code = NotFound desc = could not find container \"a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a\": container with ID starting with a4032aa471f017bf7a369f2d3cf26d5acf0dcce005bd9e5d9867e5f877033c9a not found: ID does not exist"
Oct 31 16:09:08.278810 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.278786    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 16:09:08.313587 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.313543    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\" (UniqueName: \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\") pod \"c7b09986-6611-4d04-9706-2b4c291c9720\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") "
Oct 31 16:09:08.313748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.313604    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\") pod \"c7b09986-6611-4d04-9706-2b4c291c9720\" (UID: \"c7b09986-6611-4d04-9706-2b4c291c9720\") "
Oct 31 16:09:08.314013 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.313982    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume" pod "c7b09986-6611-4d04-9706-2b4c291c9720" (UID: "c7b09986-6611-4d04-9706-2b4c291c9720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:08.315806 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.315782    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9" pod "c7b09986-6611-4d04-9706-2b4c291c9720" (UID: "c7b09986-6611-4d04-9706-2b4c291c9720"). InnerVolumeSpecName "kube-api-access-x72d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:08.415076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415040    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\" (UniqueName: \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\") pod \"9a398640-2695-466c-a576-8d241f12838f\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") "
Oct 31 16:09:08.415276 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415085    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\") pod \"9a398640-2695-466c-a576-8d241f12838f\" (UID: \"9a398640-2695-466c-a576-8d241f12838f\") "
Oct 31 16:09:08.415500 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415472    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume" pod "9a398640-2695-466c-a576-8d241f12838f" (UID: "9a398640-2695-466c-a576-8d241f12838f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:08.415647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415585    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a398640-2695-466c-a576-8d241f12838f-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:08.415647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415609    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x72d9\" (UniqueName: \"kubernetes.io/projected/c7b09986-6611-4d04-9706-2b4c291c9720-kube-api-access-x72d9\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:08.415647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.415619    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b09986-6611-4d04-9706-2b4c291c9720-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:08.417388 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.417367    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt" pod "9a398640-2695-466c-a576-8d241f12838f" (UID: "9a398640-2695-466c-a576-8d241f12838f"). InnerVolumeSpecName "kube-api-access-7xkvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:08.516625 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.516603    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xkvt\" (UniqueName: \"kubernetes.io/projected/9a398640-2695-466c-a576-8d241f12838f-kube-api-access-7xkvt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:08.635317 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.635284    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 16:09:08.641635 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:08.641605    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-31dff8-79579bc785-9w4wd"]
Oct 31 16:09:09.222217 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.222129    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"
Oct 31 16:09:09.222217 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.222167    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772" event={"ID":"9a398640-2695-466c-a576-8d241f12838f","Type":"ContainerDied","Data":"1de00a8909b9bb1fb104dff3cdb1de4aeeca5ec5083ec9fb81a109a44e6b386d"}
Oct 31 16:09:09.222767 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.222226    4549 scope.go:122] "RemoveContainer" containerID="c35b42a95a74622867ecf86f5e904f9e9f65d5b83bddcc01b9cc26f1d45d2dd3"
Oct 31 16:09:09.385791 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.385752    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 16:09:09.418914 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.418881    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" path="/var/lib/kubelet/pods/c7b09986-6611-4d04-9706-2b4c291c9720/volumes"
Oct 31 16:09:09.435711 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:09.435683    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-31dff8-67cb494d87-jc772"]
Oct 31 16:09:11.418245 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:11.418172    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a398640-2695-466c-a576-8d241f12838f" path="/var/lib/kubelet/pods/9a398640-2695-466c-a576-8d241f12838f/volumes"
Oct 31 16:09:16.560142 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:16.560106    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 16:09:21.382375 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.382337    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 16:09:21.382979 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.382547    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" podUID="03bb4eea-047b-46ff-8ee4-835d2e67c663" containerName="maas-api" containerID="cri-o://8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302" gracePeriod=30
Oct 31 16:09:21.560994 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.560945    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing" containerID="cri-o://1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75" gracePeriod=25
Oct 31 16:09:21.659661 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.659638    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 16:09:21.751713 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.751679    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\") pod \"03bb4eea-047b-46ff-8ee4-835d2e67c663\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") "
Oct 31 16:09:21.751920 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.751765    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") pod \"03bb4eea-047b-46ff-8ee4-835d2e67c663\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") "
Oct 31 16:09:21.751920 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.751906    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\") pod \"03bb4eea-047b-46ff-8ee4-835d2e67c663\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") "
Oct 31 16:09:21.752044 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.751940    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\" (UniqueName: \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\") pod \"03bb4eea-047b-46ff-8ee4-835d2e67c663\" (UID: \"03bb4eea-047b-46ff-8ee4-835d2e67c663\") "
Oct 31 16:09:21.752044 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.752020    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca" pod "03bb4eea-047b-46ff-8ee4-835d2e67c663" (UID: "03bb4eea-047b-46ff-8ee4-835d2e67c663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:21.752273 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.752249    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03bb4eea-047b-46ff-8ee4-835d2e67c663-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:21.763729 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.763667    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs" pod "03bb4eea-047b-46ff-8ee4-835d2e67c663" (UID: "03bb4eea-047b-46ff-8ee4-835d2e67c663"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:09:21.767955 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.764262    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq" pod "03bb4eea-047b-46ff-8ee4-835d2e67c663" (UID: "03bb4eea-047b-46ff-8ee4-835d2e67c663"). InnerVolumeSpecName "kube-api-access-ldndq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:21.767955 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.767116    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls" pod "03bb4eea-047b-46ff-8ee4-835d2e67c663" (UID: "03bb4eea-047b-46ff-8ee4-835d2e67c663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:09:21.853681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.853651    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:21.853681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.853677    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03bb4eea-047b-46ff-8ee4-835d2e67c663-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:21.853681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:21.853687    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldndq\" (UniqueName: \"kubernetes.io/projected/03bb4eea-047b-46ff-8ee4-835d2e67c663-kube-api-access-ldndq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:22.307789 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.307757    4549 generic.go:356] "Generic (PLEG): container finished" podID="03bb4eea-047b-46ff-8ee4-835d2e67c663" containerID="8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302" exitCode=0
Oct 31 16:09:22.307992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.307817    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"
Oct 31 16:09:22.307992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.307846    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" event={"ID":"03bb4eea-047b-46ff-8ee4-835d2e67c663","Type":"ContainerDied","Data":"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"}
Oct 31 16:09:22.307992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.307901    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2" event={"ID":"03bb4eea-047b-46ff-8ee4-835d2e67c663","Type":"ContainerDied","Data":"e18309b0b3b429e5b479c51631e4d367352c30e54dcb332ec1bc38f185d11aa0"}
Oct 31 16:09:22.307992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.307928    4549 scope.go:122] "RemoveContainer" containerID="8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"
Oct 31 16:09:22.324185 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.324157    4549 scope.go:122] "RemoveContainer" containerID="8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"
Oct 31 16:09:22.324563 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:22.324539    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302\": container with ID starting with 8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302 not found: ID does not exist" containerID="8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"
Oct 31 16:09:22.324663 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.324593    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302"} err="failed to get container status \"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302\": rpc error: code = NotFound desc = could not find container \"8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302\": container with ID starting with 8129d2827103cf820e8ca034449889058098c6a50c457bcbd823fe1dca280302 not found: ID does not exist"
Oct 31 16:09:22.469884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.469852    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 16:09:22.473609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:22.473585    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-140c48-5db78c848d-f4xh2"]
Oct 31 16:09:23.418024 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:23.417991    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bb4eea-047b-46ff-8ee4-835d2e67c663" path="/var/lib/kubelet/pods/03bb4eea-047b-46ff-8ee4-835d2e67c663/volumes"
Oct 31 16:09:24.952144 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:24.952104    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 16:09:24.953021 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:24.952986    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy" containerID="cri-o://e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec" gracePeriod=30
Oct 31 16:09:30.079378 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.079348    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:09:30.214160 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.214134    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:09:30.342953 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.342913    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343175 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343041    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343175 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343080    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343175 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343112    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343175 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343141    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343412 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343192    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343412 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343231    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343412 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343263    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343412 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343333    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\") pod \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\" (UID: \"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6\") "
Oct 31 16:09:30.343694 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343329    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:30.343694 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.343519    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:30.344150 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.344043    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:30.344150 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.344115    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:30.344347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.344222    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:30.345610 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.345550    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:30.346226 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.346207    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "kube-api-access-znlgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:30.346318 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.346234    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:30.346658 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.346641    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo" pod "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" (UID: "870ab6ea-5aee-4e04-b015-8f99ad0ce2d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:09:30.362504 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.362473    4549 generic.go:356] "Generic (PLEG): container finished" podID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerID="e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec" exitCode=0
Oct 31 16:09:30.362673 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.362561    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"
Oct 31 16:09:30.362673 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.362546    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerDied","Data":"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"}
Oct 31 16:09:30.362791 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.362675    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd" event={"ID":"870ab6ea-5aee-4e04-b015-8f99ad0ce2d6","Type":"ContainerDied","Data":"6b34795fdb68c613cfe49267fc10b6f5b11cf2452e643f1f7c337e5c256f958f"}
Oct 31 16:09:30.362791 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.362710    4549 scope.go:122] "RemoveContainer" containerID="e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"
Oct 31 16:09:30.366550 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.366529    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 16:09:30.374077 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.374057    4549 scope.go:122] "RemoveContainer" containerID="f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"
Oct 31 16:09:30.383494 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.383411    4549 scope.go:122] "RemoveContainer" containerID="e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"
Oct 31 16:09:30.383782 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:30.383764    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec\": container with ID starting with e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec not found: ID does not exist" containerID="e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"
Oct 31 16:09:30.383870 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.383799    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec"} err="failed to get container status \"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec\": rpc error: code = NotFound desc = could not find container \"e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec\": container with ID starting with e4ffd6001e1aa6624a3e1a3c54bda2281896ee4b3e245639b8859f880b3bb5ec not found: ID does not exist"
Oct 31 16:09:30.383870 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.383822    4549 scope.go:122] "RemoveContainer" containerID="f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"
Oct 31 16:09:30.384165 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:30.384145    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513\": container with ID starting with f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513 not found: ID does not exist" containerID="f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"
Oct 31 16:09:30.384218 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.384175    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513"} err="failed to get container status \"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513\": rpc error: code = NotFound desc = could not find container \"f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513\": container with ID starting with f886ca41d403dae280abdc1924750104f479091a607ce7eb93111a3339d08513 not found: ID does not exist"
Oct 31 16:09:30.444460 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444430    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444460 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444455    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444460 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444466    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444474    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444485    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444494    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlgn\" (UniqueName: \"kubernetes.io/projected/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-kube-api-access-znlgn\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444504    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444512    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.444748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.444521    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:30.614684 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.614592    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 16:09:30.614930 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:30.614918    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-140c48-openshift-default-67444799f7-r5rxd"]
Oct 31 16:09:31.417946 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:31.417912    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" path="/var/lib/kubelet/pods/870ab6ea-5aee-4e04-b015-8f99ad0ce2d6/volumes"
Oct 31 16:09:37.013468 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:37.013421    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:09:40.079468 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:40.079430    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:09:41.819682 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:41.819658    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 16:09:41.959269 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:41.959181    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\" (UniqueName: \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\") pod \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") "
Oct 31 16:09:41.959450 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:41.959326    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\") pod \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\" (UID: \"8e9140fa-f90a-4c7c-a07a-f5495534cae2\") "
Oct 31 16:09:41.959831 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:41.959801    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume" pod "8e9140fa-f90a-4c7c-a07a-f5495534cae2" (UID: "8e9140fa-f90a-4c7c-a07a-f5495534cae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:09:41.961607 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:41.961551    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd" pod "8e9140fa-f90a-4c7c-a07a-f5495534cae2" (UID: "8e9140fa-f90a-4c7c-a07a-f5495534cae2"). InnerVolumeSpecName "kube-api-access-8rvqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:42.060677 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.060644    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9140fa-f90a-4c7c-a07a-f5495534cae2-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:42.060677 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.060673    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvqd\" (UniqueName: \"kubernetes.io/projected/8e9140fa-f90a-4c7c-a07a-f5495534cae2-kube-api-access-8rvqd\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:42.437261 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.437221    4549 generic.go:356] "Generic (PLEG): container finished" podID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerID="1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75" exitCode=0
Oct 31 16:09:42.437443 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.437280    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"
Oct 31 16:09:42.437443 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.437298    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" event={"ID":"8e9140fa-f90a-4c7c-a07a-f5495534cae2","Type":"ContainerDied","Data":"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"}
Oct 31 16:09:42.437443 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.437356    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr" event={"ID":"8e9140fa-f90a-4c7c-a07a-f5495534cae2","Type":"ContainerDied","Data":"785944f0960d992d8ca8e89affdce4f1435b0d51840ce6e3019378f4397d0406"}
Oct 31 16:09:42.437443 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.437380    4549 scope.go:122] "RemoveContainer" containerID="1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"
Oct 31 16:09:42.447003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.446982    4549 scope.go:122] "RemoveContainer" containerID="1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"
Oct 31 16:09:42.447322 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:42.447293    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75\": container with ID starting with 1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75 not found: ID does not exist" containerID="1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"
Oct 31 16:09:42.447404 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.447335    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75"} err="failed to get container status \"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75\": rpc error: code = NotFound desc = could not find container \"1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75\": container with ID starting with 1dc5b33f380d9bd35f8176a453ee8e278ce08ae15acc548cf6296428cc1b1a75 not found: ID does not exist"
Oct 31 16:09:42.621718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.621683    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 16:09:42.678717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:42.678685    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-140c48-69f8bb6879-969dr"]
Oct 31 16:09:43.417799 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:43.417766    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" path="/var/lib/kubelet/pods/8e9140fa-f90a-4c7c-a07a-f5495534cae2/volumes"
Oct 31 16:09:45.387792 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.387751    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" containerID="cri-o://9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3" gracePeriod=45
Oct 31 16:09:45.473035 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.473000    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused"
Oct 31 16:09:45.628609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.628562    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 16:09:45.798064 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798032    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798250 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798089    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798250 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798165    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\" (UniqueName: \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798250 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798219    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\" (UniqueName: \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798253    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798308    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\") pod \"827ff8fc-d174-4f66-838f-044a9201652e\" (UID: \"827ff8fc-d174-4f66-838f-044a9201652e\") "
Oct 31 16:09:45.798389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798303    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:45.798389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798371    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:45.798634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.798596    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:45.799231 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.799109    4549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-model-cache\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:45.799231 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.799134    4549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-home\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:45.799231 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.799151    4549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-tmp-dir\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:45.800830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.800807    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:09:45.800929 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.800892    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:09:45.801340 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.801317    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6" pod "827ff8fc-d174-4f66-838f-044a9201652e" (UID: "827ff8fc-d174-4f66-838f-044a9201652e"). InnerVolumeSpecName "kube-api-access-kj9m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:09:45.900419 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.900387    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9m6\" (UniqueName: \"kubernetes.io/projected/827ff8fc-d174-4f66-838f-044a9201652e-kube-api-access-kj9m6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:45.900419 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.900413    4549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/827ff8fc-d174-4f66-838f-044a9201652e-tls-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:45.900419 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:45.900422    4549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/827ff8fc-d174-4f66-838f-044a9201652e-dshm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:09:46.461367 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.461331    4549 generic.go:356] "Generic (PLEG): container finished" podID="827ff8fc-d174-4f66-838f-044a9201652e" containerID="9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3" exitCode=0
Oct 31 16:09:46.461876 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.461405    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"
Oct 31 16:09:46.461876 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.461414    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" event={"ID":"827ff8fc-d174-4f66-838f-044a9201652e","Type":"ContainerDied","Data":"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"}
Oct 31 16:09:46.461876 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.461459    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v" event={"ID":"827ff8fc-d174-4f66-838f-044a9201652e","Type":"ContainerDied","Data":"8adce29fff2795c946822a4a03c0d27c645da39233cd263f6cc7dadcd41249e6"}
Oct 31 16:09:46.461876 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.461475    4549 scope.go:122] "RemoveContainer" containerID="9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"
Oct 31 16:09:46.470955 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.470938    4549 scope.go:122] "RemoveContainer" containerID="9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"
Oct 31 16:09:46.471214 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:09:46.471195    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3\": container with ID starting with 9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3 not found: ID does not exist" containerID="9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"
Oct 31 16:09:46.471262 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.471223    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3"} err="failed to get container status \"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3\": rpc error: code = NotFound desc = could not find container \"9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3\": container with ID starting with 9b8605efbbb4362a5a66d0b78dc97c4145d1c4a522cff0c82f030df08b67eda3 not found: ID does not exist"
Oct 31 16:09:46.629092 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.629059    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 16:09:46.678506 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:46.678472    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/distinct-2-w0-56bc96-kserve-7b4b4bff85-k9k4v"]
Oct 31 16:09:47.419159 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:09:47.419126    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827ff8fc-d174-4f66-838f-044a9201652e" path="/var/lib/kubelet/pods/827ff8fc-d174-4f66-838f-044a9201652e/volumes"
Oct 31 16:10:07.012966 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:07.012930    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:10:10.303043 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:10.302991    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 16:10:10.304156 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:10.304081    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" containerID="cri-o://15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f" gracePeriod=30
Oct 31 16:10:11.857626 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:11.857596    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z7xph_fe94f527-0b08-478a-baa0-4f92659bf5d6/manager/0.log"
Oct 31 16:10:12.180633 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:12.180521    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/0.log"
Oct 31 16:10:12.696980 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:12.696949    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b7wgz_b301180c-09ce-4d9b-abf3-45e80856e298/manager/0.log"
Oct 31 16:10:14.030138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:14.030103    4549 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:10:14.030533 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:14.030158    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:10:15.553217 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.553188    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 16:10:15.630614 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.630517    4549 generic.go:356] "Generic (PLEG): container finished" podID="bc710aa4-1d7a-4421-a876-e5a482292056" containerID="15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f" exitCode=0
Oct 31 16:10:15.630614 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.630561    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" event={"ID":"bc710aa4-1d7a-4421-a876-e5a482292056","Type":"ContainerDied","Data":"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"}
Oct 31 16:10:15.630834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.630631    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg" event={"ID":"bc710aa4-1d7a-4421-a876-e5a482292056","Type":"ContainerDied","Data":"ceb42442b71043bffb20da64f04e73b9e77df0c705bd4601a3cf5a596d702fbe"}
Oct 31 16:10:15.630834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.630653    4549 scope.go:122] "RemoveContainer" containerID="15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"
Oct 31 16:10:15.630834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.630584    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"
Oct 31 16:10:15.639704 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.639683    4549 scope.go:122] "RemoveContainer" containerID="15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"
Oct 31 16:10:15.639950 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:15.639922    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f\": container with ID starting with 15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f not found: ID does not exist" containerID="15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"
Oct 31 16:10:15.640022 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.639957    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f"} err="failed to get container status \"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f\": rpc error: code = NotFound desc = could not find container \"15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f\": container with ID starting with 15bb8baaaf75f0609c004cdfd44ca6a6fd9a714086ac7460db0aee1148f6ba2f not found: ID does not exist"
Oct 31 16:10:15.654793 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654770    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.654905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654813    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.654905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654847    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.654905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654890    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655038 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654921    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655038 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.654961    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655038 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655004    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655203 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655045    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655203 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655076    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:15.655203 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655080    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\") pod \"bc710aa4-1d7a-4421-a876-e5a482292056\" (UID: \"bc710aa4-1d7a-4421-a876-e5a482292056\") "
Oct 31 16:10:15.655370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655213    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:15.655370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655297    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:15.655370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655316    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:15.655540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655371    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:10:15.655540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655487    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.655540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655509    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.655540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655524    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc710aa4-1d7a-4421-a876-e5a482292056-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.655540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655538    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.655808 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.655554    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.657253 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.657230    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:15.657428 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.657404    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:10:15.657906 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.657883    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:10:15.657906 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.657892    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q" pod "bc710aa4-1d7a-4421-a876-e5a482292056" (UID: "bc710aa4-1d7a-4421-a876-e5a482292056"). InnerVolumeSpecName "kube-api-access-vdd7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:10:15.735353 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.735307    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fc9km_fac77ca4-d950-4750-9ed3-3c8550b6549e/manager/0.log"
Oct 31 16:10:15.756609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.756558    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc710aa4-1d7a-4421-a876-e5a482292056-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.756609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.756604    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.756834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.756618    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc710aa4-1d7a-4421-a876-e5a482292056-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:15.756834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:15.756630    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdd7q\" (UniqueName: \"kubernetes.io/projected/bc710aa4-1d7a-4421-a876-e5a482292056-kube-api-access-vdd7q\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:16.074871 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:16.074835    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 16:10:16.083948 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:16.083921    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-5wlpg"]
Oct 31 16:10:17.249088 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:17.249054    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-dff98979c-sr98m_4f43ebaf-2e1d-4f28-9e85-71f708747990/manager/0.log"
Oct 31 16:10:17.417791 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:17.417761    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" path="/var/lib/kubelet/pods/bc710aa4-1d7a-4421-a876-e5a482292056/volumes"
Oct 31 16:10:23.131317 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:23.131283    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 16:10:23.131809 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:23.131477    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy" containerID="cri-o://f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d" gracePeriod=30
Oct 31 16:10:28.380023 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.380000    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 16:10:28.473539 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473462    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473539 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473499    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473539 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473523    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473562    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473624    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473690    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.473853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473794    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:28.473853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473798    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.474151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473905    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.474151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.473943    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\") pod \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\" (UID: \"8d90fda0-05d1-4eda-83d4-bd5a23d2879a\") "
Oct 31 16:10:28.474261 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474055    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:10:28.474261 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474216    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:28.474403 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474382    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.474655 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474404    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:28.474655 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474410    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.474655 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474626    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.474899 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.474877    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:28.476225 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.476196    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:10:28.476348 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.476284    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:10:28.476348 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.476304    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "kube-api-access-q5kpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:10:28.476348 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.476308    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy" pod "8d90fda0-05d1-4eda-83d4-bd5a23d2879a" (UID: "8d90fda0-05d1-4eda-83d4-bd5a23d2879a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:10:28.575131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575097    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.575131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575126    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.575131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575138    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.575378 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575151    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kpk\" (UniqueName: \"kubernetes.io/projected/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-kube-api-access-q5kpk\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.575378 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575167    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.575378 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.575179    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d90fda0-05d1-4eda-83d4-bd5a23d2879a-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:10:28.702772 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.702741    4549 generic.go:356] "Generic (PLEG): container finished" podID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerID="f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d" exitCode=0
Oct 31 16:10:28.702772 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.702777    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" event={"ID":"8d90fda0-05d1-4eda-83d4-bd5a23d2879a","Type":"ContainerDied","Data":"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"}
Oct 31 16:10:28.703027 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.702790    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"
Oct 31 16:10:28.703027 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.702813    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d" event={"ID":"8d90fda0-05d1-4eda-83d4-bd5a23d2879a","Type":"ContainerDied","Data":"51ae8b85d4e093be4625a116d45a05c02e444066782d305595b63945d43933f9"}
Oct 31 16:10:28.703027 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.702833    4549 scope.go:122] "RemoveContainer" containerID="f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"
Oct 31 16:10:28.713215 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.713198    4549 scope.go:122] "RemoveContainer" containerID="f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"
Oct 31 16:10:28.713475 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:28.713446    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d\": container with ID starting with f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d not found: ID does not exist" containerID="f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"
Oct 31 16:10:28.713539 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.713477    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d"} err="failed to get container status \"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d\": rpc error: code = NotFound desc = could not find container \"f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d\": container with ID starting with f05b207cf83344dfa7abf470927ced93b4dbce29fa7b9bc38595ad99b760637d not found: ID does not exist"
Oct 31 16:10:28.888217 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.888182    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 16:10:28.891021 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:28.891000    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-lbl5d"]
Oct 31 16:10:29.418176 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:29.418146    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" path="/var/lib/kubelet/pods/8d90fda0-05d1-4eda-83d4-bd5a23d2879a/volumes"
Oct 31 16:10:32.891153 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891122    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891461    4549 cpu_manager.go:414] "containerMap: removing container" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891472    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891489    4549 cpu_manager.go:414] "containerMap: removing container" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891494    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891504    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891509    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891517    4549 cpu_manager.go:414] "containerMap: removing container" podUID="a6735b72-4477-4929-84ba-a271819780e4" containerName="main"
Oct 31 16:10:32.891524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891522    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6735b72-4477-4929-84ba-a271819780e4" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891531    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891535    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891543    4549 cpu_manager.go:414] "containerMap: removing container" podUID="92463f88-bab2-4936-9c20-350c03b92071" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891547    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92463f88-bab2-4936-9c20-350c03b92071" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891557    4549 cpu_manager.go:414] "containerMap: removing container" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891562    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891581    4549 cpu_manager.go:414] "containerMap: removing container" podUID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891585    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891597    4549 cpu_manager.go:414] "containerMap: removing container" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891601    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891617    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891622    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891627    4549 cpu_manager.go:414] "containerMap: removing container" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891632    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891638    4549 cpu_manager.go:414] "containerMap: removing container" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891642    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891649    4549 cpu_manager.go:414] "containerMap: removing container" podUID="03bb4eea-047b-46ff-8ee4-835d2e67c663" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891653    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bb4eea-047b-46ff-8ee4-835d2e67c663" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891661    4549 cpu_manager.go:414] "containerMap: removing container" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891665    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891721    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891730    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="870ab6ea-5aee-4e04-b015-8f99ad0ce2d6" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891737    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d90fda0-05d1-4eda-83d4-bd5a23d2879a" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891742    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6735b72-4477-4929-84ba-a271819780e4" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891749    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9140fa-f90a-4c7c-a07a-f5495534cae2" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891755    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="92463f88-bab2-4936-9c20-350c03b92071" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891761    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a398640-2695-466c-a576-8d241f12838f" containerName="payload-pre-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891766    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9dae8a-d3e1-49f4-b540-5055b58aefd5" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891771    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc710aa4-1d7a-4421-a876-e5a482292056" containerName="istio-proxy"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891776    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="1859f7ae-117c-435f-8cf4-83d1d456e128" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891781    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b09986-6611-4d04-9706-2b4c291c9720" containerName="payload-processing"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891786    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="827ff8fc-d174-4f66-838f-044a9201652e" containerName="main"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891791    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bb4eea-047b-46ff-8ee4-835d2e67c663" containerName="maas-api"
Oct 31 16:10:32.891830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.891795    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="7107cbd7-f4e5-47b8-842e-95c89d8d952e" containerName="istio-proxy"
Oct 31 16:10:32.896186 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.896170    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:32.951550 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.951516    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:10:32.998045 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.998011    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9e4bd6cf-serving-cert\""
Oct 31 16:10:32.998045 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.998026    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:10:32.998045 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.998050    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 16:10:32.998352 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.998083    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:10:32.998352 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:32.998131    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:10:33.018673 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.018641    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.018848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.018691    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.018848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.018815    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.018971 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.018863    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvsh\" (UniqueName: \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.119554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.119520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.119554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.119554    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvsh\" (UniqueName: \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.119820 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.119654    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.119820 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:33.119680    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-serving-cert: secret "maas-api-e2e-ait-9e4bd6cf-serving-cert" not found
Oct 31 16:10:33.119820 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.119720    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.119820 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:33.119756    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls podName:c74cd599-378e-4348-868b-1675e9600cc6 nodeName:}" failed. No retries permitted until 2026-10-31 16:10:33.619734202 +0000 UTC m=+5074.346015160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls") pod "maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" (UID: "c74cd599-378e-4348-868b-1675e9600cc6") : secret "maas-api-e2e-ait-9e4bd6cf-serving-cert" not found
Oct 31 16:10:33.120371 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.120341    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.122328 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.122306    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.177222 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.177155    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvsh\" (UniqueName: \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.625593 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.625535    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.628272 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.628241    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") pod \"maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:33.808752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:33.808709    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:34.182036 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:34.182011    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:10:34.183673 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:10:34.183637    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74cd599_378e_4348_868b_1675e9600cc6.slice/crio-99c085f19540273277dc3874f82a2b2fc1b555e8622cfe49b67d67be7f884ac0 WatchSource:0}: Error finding container 99c085f19540273277dc3874f82a2b2fc1b555e8622cfe49b67d67be7f884ac0: Status 404 returned error can't find the container with id 99c085f19540273277dc3874f82a2b2fc1b555e8622cfe49b67d67be7f884ac0
Oct 31 16:10:34.185092 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:34.185073    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:10:34.751814 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:34.751776    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" event={"ID":"c74cd599-378e-4348-868b-1675e9600cc6","Type":"ContainerStarted","Data":"99c085f19540273277dc3874f82a2b2fc1b555e8622cfe49b67d67be7f884ac0"}
Oct 31 16:10:35.759367 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:35.759332    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" event={"ID":"c74cd599-378e-4348-868b-1675e9600cc6","Type":"ContainerStarted","Data":"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"}
Oct 31 16:10:35.759873 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:35.759441    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:10:35.874472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:35.874417    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" podStartSLOduration=3.005634704 podStartE2EDuration="3.874401655s" totalImagesPullingTime="868.766951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:10:35.873712966 +0000 UTC m=+5076.599993945" watchObservedRunningTime="2026-10-31 16:10:35.874401655 +0000 UTC m=+5076.600682632"
Oct 31 16:10:37.012221 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:10:37.012184    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:10:41.770588 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:10:41.770501    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:11:03.281725 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:03.281695    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:11:03.438559 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:03.438529    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:11:07.012162 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:07.012119    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:11:09.238030 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.237988    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:09.245048 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.245021    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.296893 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.296862    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e4bd6cf-openshift-default-dockercfg-b58z4\""
Oct 31 16:11:09.348742 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.348704    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.348742 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.348742    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.348961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.348862    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.348961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.348892    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574mm\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.348961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.348924    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.349082 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.349010    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.349082 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.349031    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.349082 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.349048    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:09.349082 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.349065    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.349234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.349102    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.450551 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450456    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.450551 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450512    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450605    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450640    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574mm\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450687    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.450742    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451083    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451355 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451131    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451355 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451159    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451835 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451805    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.451961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451900    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.452040 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.451961    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.452040 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.452010    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.452376 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.452351    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.454471 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.454449    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.456068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.456045    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.507341 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.507272    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.507517 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.507451    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574mm\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.558662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.558632    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:09.980389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.980362    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:09.982027 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:11:09.982000    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1c0578_eb92_4770_b75e_be36446122a8.slice/crio-a12b063ce56c133400ce09c798f2f10392f221b0a14dc7fc2f090e415ab19e2c WatchSource:0}: Error finding container a12b063ce56c133400ce09c798f2f10392f221b0a14dc7fc2f090e415ab19e2c: Status 404 returned error can't find the container with id a12b063ce56c133400ce09c798f2f10392f221b0a14dc7fc2f090e415ab19e2c
Oct 31 16:11:09.984111 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.984079    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:09.984201 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.984142    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:09.984201 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:09.984169    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:10.951555 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:10.951519    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" event={"ID":"3c1c0578-eb92-4770-b75e-be36446122a8","Type":"ContainerStarted","Data":"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"}
Oct 31 16:11:10.952042 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:10.951603    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" event={"ID":"3c1c0578-eb92-4770-b75e-be36446122a8","Type":"ContainerStarted","Data":"a12b063ce56c133400ce09c798f2f10392f221b0a14dc7fc2f090e415ab19e2c"}
Oct 31 16:11:11.167533 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:11.167474    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podStartSLOduration=2.167459778 podStartE2EDuration="2.167459778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:11.117026011 +0000 UTC m=+5111.843306990" watchObservedRunningTime="2026-10-31 16:11:11.167459778 +0000 UTC m=+5111.893740756"
Oct 31 16:11:11.559692 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:11.559662    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:11.560854 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:11.560827    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 16:11:11.560980 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:11.560871    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 16:11:12.559146 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:12.559111    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 16:11:12.559600 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:12.559165    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 16:11:13.559374 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:13.559340    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 16:11:13.559827 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:13.559392    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 16:11:14.559373 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:14.559338    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 16:11:14.559839 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:14.559392    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 16:11:15.559480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:15.559442    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 31 16:11:15.559910 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:15.559496    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 31 16:11:16.563447 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:16.563416    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:16.563967 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:16.563823    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:16.564549 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:16.564531    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:37.011291 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:37.011249    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:11:39.216108 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.216064    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:11:39.216551 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.216289    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" podUID="c74cd599-378e-4348-868b-1675e9600cc6" containerName="maas-api" containerID="cri-o://67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08" gracePeriod=30
Oct 31 16:11:39.466023 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.465995    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:39.469844 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.469786    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.472646 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.472628    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:11:39.492378 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.492351    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:39.492700 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.492688    4549 cpu_manager.go:414] "containerMap: removing container" podUID="c74cd599-378e-4348-868b-1675e9600cc6" containerName="maas-api"
Oct 31 16:11:39.492758 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.492702    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74cd599-378e-4348-868b-1675e9600cc6" containerName="maas-api"
Oct 31 16:11:39.492830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.492818    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74cd599-378e-4348-868b-1675e9600cc6" containerName="maas-api"
Oct 31 16:11:39.495753 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.495738    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.527135 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.527105    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:39.584003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.583962    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:39.648805 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.648779    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") pod \"c74cd599-378e-4348-868b-1675e9600cc6\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") "
Oct 31 16:11:39.648990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.648848    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\" (UniqueName: \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\") pod \"c74cd599-378e-4348-868b-1675e9600cc6\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") "
Oct 31 16:11:39.648990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.648881    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\") pod \"c74cd599-378e-4348-868b-1675e9600cc6\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") "
Oct 31 16:11:39.648990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.648947    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\") pod \"c74cd599-378e-4348-868b-1675e9600cc6\" (UID: \"c74cd599-378e-4348-868b-1675e9600cc6\") "
Oct 31 16:11:39.649179 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649058    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.649179 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649099    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29l57\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649179 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649124    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649350 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649221    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649350 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649290    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.649350 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649314    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.649515 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649362    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.649515 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649441    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca" pod "c74cd599-378e-4348-868b-1675e9600cc6" (UID: "c74cd599-378e-4348-868b-1675e9600cc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:11:39.649515 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649477    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649705 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649545    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649826 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649696    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8sm\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.649882 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649865    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649923 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649899    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.649982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.649928    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.650037 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650018    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.650076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650045    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.650126 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650075    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.650126 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650101    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.650226 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650172    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.650321 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.650303    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74cd599-378e-4348-868b-1675e9600cc6-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:39.651368 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.651349    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs" pod "c74cd599-378e-4348-868b-1675e9600cc6" (UID: "c74cd599-378e-4348-868b-1675e9600cc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:11:39.651688 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.651674    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls" pod "c74cd599-378e-4348-868b-1675e9600cc6" (UID: "c74cd599-378e-4348-868b-1675e9600cc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:11:39.651752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.651713    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh" pod "c74cd599-378e-4348-868b-1675e9600cc6" (UID: "c74cd599-378e-4348-868b-1675e9600cc6"). InnerVolumeSpecName "kube-api-access-zrvsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:39.750987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.750905    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.750987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.750942    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.750987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.750964    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.750987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.750984    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751018    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29l57\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751047    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751070    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751099    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751126    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751163    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751195    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751247    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751290    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8sm\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751358 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751331    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751373    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751387    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751418    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751471    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751612    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751654    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751692    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751720    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751722    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.751998 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.751780    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752035    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752074    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752074    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752318    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752353    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvsh\" (UniqueName: \"kubernetes.io/projected/c74cd599-378e-4348-868b-1675e9600cc6-kube-api-access-zrvsh\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752404    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:39.752505 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.752422    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74cd599-378e-4348-868b-1675e9600cc6-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:39.753941 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.753917    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.754338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.754321    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.754439 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.754424    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.754481 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.754425    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.808107 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.808078    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8sm\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.808397 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.808375    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29l57\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:39.808455 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.808382    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:39.808837 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:39.808815    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:40.081488 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.081453    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:40.099777 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.099745    4549 generic.go:356] "Generic (PLEG): container finished" podID="c74cd599-378e-4348-868b-1675e9600cc6" containerID="67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08" exitCode=0
Oct 31 16:11:40.099900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.099827    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" event={"ID":"c74cd599-378e-4348-868b-1675e9600cc6","Type":"ContainerDied","Data":"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"}
Oct 31 16:11:40.099900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.099855    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"
Oct 31 16:11:40.099900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.099886    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p" event={"ID":"c74cd599-378e-4348-868b-1675e9600cc6","Type":"ContainerDied","Data":"99c085f19540273277dc3874f82a2b2fc1b555e8622cfe49b67d67be7f884ac0"}
Oct 31 16:11:40.100089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.099918    4549 scope.go:122] "RemoveContainer" containerID="67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"
Oct 31 16:11:40.107924 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.107725    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:40.110716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.110694    4549 scope.go:122] "RemoveContainer" containerID="67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"
Oct 31 16:11:40.111167 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:40.111146    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08\": container with ID starting with 67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08 not found: ID does not exist" containerID="67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"
Oct 31 16:11:40.111258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.111181    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08"} err="failed to get container status \"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08\": rpc error: code = NotFound desc = could not find container \"67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08\": container with ID starting with 67d97e98b167dff926e6fb13f2e7ba78d6b768b69a97e44c54c6363064b4af08 not found: ID does not exist"
Oct 31 16:11:40.263916 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.263882    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:11:40.265952 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.265933    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9e4bd6cf-6d9779c5ff-mlk7p"]
Oct 31 16:11:40.518955 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.518929    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:40.519903 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:11:40.519730    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4953c2db_379c_410d_aea7_732463aa4481.slice/crio-7778abe34cd1aab905ad110b76685fa11616b7b26be55b295929bfab00c2151b WatchSource:0}: Error finding container 7778abe34cd1aab905ad110b76685fa11616b7b26be55b295929bfab00c2151b: Status 404 returned error can't find the container with id 7778abe34cd1aab905ad110b76685fa11616b7b26be55b295929bfab00c2151b
Oct 31 16:11:40.521309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.521287    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:40.521945 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:11:40.521918    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb590ab7c_1c17_447c_adb2_7dab068f729a.slice/crio-f18712af7615182f7259365ecee6294b193f5724b5bb918e406e224824a38445 WatchSource:0}: Error finding container f18712af7615182f7259365ecee6294b193f5724b5bb918e406e224824a38445: Status 404 returned error can't find the container with id f18712af7615182f7259365ecee6294b193f5724b5bb918e406e224824a38445
Oct 31 16:11:40.522506 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.522468    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:40.522627 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.522561    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:40.522760 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.522651    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:40.524070 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.524048    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:40.524140 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.524098    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:40.524140 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:40.524128    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:11:41.125602 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.123095    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" event={"ID":"4953c2db-379c-410d-aea7-732463aa4481","Type":"ContainerStarted","Data":"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"}
Oct 31 16:11:41.125602 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.123160    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" event={"ID":"4953c2db-379c-410d-aea7-732463aa4481","Type":"ContainerStarted","Data":"7778abe34cd1aab905ad110b76685fa11616b7b26be55b295929bfab00c2151b"}
Oct 31 16:11:41.143609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.143419    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" event={"ID":"b590ab7c-1c17-447c-adb2-7dab068f729a","Type":"ContainerStarted","Data":"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"}
Oct 31 16:11:41.143609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.143486    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" event={"ID":"b590ab7c-1c17-447c-adb2-7dab068f729a","Type":"ContainerStarted","Data":"f18712af7615182f7259365ecee6294b193f5724b5bb918e406e224824a38445"}
Oct 31 16:11:41.257006 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.256947    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podStartSLOduration=2.256931822 podStartE2EDuration="2.256931822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:41.255514632 +0000 UTC m=+5141.981795636" watchObservedRunningTime="2026-10-31 16:11:41.256931822 +0000 UTC m=+5141.983212799"
Oct 31 16:11:41.374388 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.374331    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podStartSLOduration=2.374317726 podStartE2EDuration="2.374317726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:41.373663287 +0000 UTC m=+5142.099944267" watchObservedRunningTime="2026-10-31 16:11:41.374317726 +0000 UTC m=+5142.100598704"
Oct 31 16:11:41.460458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:41.460380    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74cd599-378e-4348-868b-1675e9600cc6" path="/var/lib/kubelet/pods/c74cd599-378e-4348-868b-1675e9600cc6/volumes"
Oct 31 16:11:42.082052 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.082011    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:42.083496 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.083465    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:11:42.083707 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.083682    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 16:11:42.114596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.112952    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:42.118594 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.115690    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 16:11:42.118594 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:42.115748    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 16:11:43.082100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:43.082063    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:11:43.082531 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:43.082119    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 16:11:43.117596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:43.116993    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 16:11:43.117596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:43.117053    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 16:11:44.082194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:44.082156    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:11:44.082669 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:44.082225    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 16:11:44.116038 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:44.116004    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 16:11:44.116320 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:44.116291    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 16:11:45.082158 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.082116    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Oct 31 16:11:45.082348 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.082185    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Oct 31 16:11:45.112606 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.112488    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 31 16:11:45.112606 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.112551    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 31 16:11:45.279045 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.278980    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:45.281653 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.279285    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy" containerID="cri-o://1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6" gracePeriod=30
Oct 31 16:11:45.282528 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.282507    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:45.282846 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.282827    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy" containerID="cri-o://fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb" gracePeriod=30
Oct 31 16:11:45.329551 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.329499    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:45.359983 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:45.359888    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" containerID="cri-o://eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b" gracePeriod=30
Oct 31 16:11:46.567085 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:46.567045    4549 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 16:11:46.567563 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:46.567127    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 16:11:49.734677 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:49.734637    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 16:11:49.735189 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:49.734849    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-dff98979c-sr98m" podUID="4f43ebaf-2e1d-4f28-9e85-71f708747990" containerName="manager" containerID="cri-o://704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f" gracePeriod=30
Oct 31 16:11:49.983067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:49.983045    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 16:11:50.086071 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.086035    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\") pod \"4f43ebaf-2e1d-4f28-9e85-71f708747990\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") "
Oct 31 16:11:50.086288 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.086130    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\" (UniqueName: \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\") pod \"4f43ebaf-2e1d-4f28-9e85-71f708747990\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") "
Oct 31 16:11:50.086288 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.086216    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\") pod \"4f43ebaf-2e1d-4f28-9e85-71f708747990\" (UID: \"4f43ebaf-2e1d-4f28-9e85-71f708747990\") "
Oct 31 16:11:50.088510 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.088477    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs" pod "4f43ebaf-2e1d-4f28-9e85-71f708747990" (UID: "4f43ebaf-2e1d-4f28-9e85-71f708747990"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:11:50.088679 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.088519    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert" pod "4f43ebaf-2e1d-4f28-9e85-71f708747990" (UID: "4f43ebaf-2e1d-4f28-9e85-71f708747990"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:11:50.088679 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.088526    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475" pod "4f43ebaf-2e1d-4f28-9e85-71f708747990" (UID: "4f43ebaf-2e1d-4f28-9e85-71f708747990"). InnerVolumeSpecName "kube-api-access-qq475". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:50.187708 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.187675    4549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-webhook-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.187708 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.187703    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f43ebaf-2e1d-4f28-9e85-71f708747990-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.187708 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.187714    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq475\" (UniqueName: \"kubernetes.io/projected/4f43ebaf-2e1d-4f28-9e85-71f708747990-kube-api-access-qq475\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.203934 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.203898    4549 generic.go:356] "Generic (PLEG): container finished" podID="4f43ebaf-2e1d-4f28-9e85-71f708747990" containerID="704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f" exitCode=0
Oct 31 16:11:50.204115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.203946    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-dff98979c-sr98m"
Oct 31 16:11:50.204115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.203980    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-sr98m" event={"ID":"4f43ebaf-2e1d-4f28-9e85-71f708747990","Type":"ContainerDied","Data":"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"}
Oct 31 16:11:50.204115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.204052    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-dff98979c-sr98m" event={"ID":"4f43ebaf-2e1d-4f28-9e85-71f708747990","Type":"ContainerDied","Data":"2d48dd817e36321fd953dfe8314ba9bcdd1603e59dd689cf5d9391c4a864ead0"}
Oct 31 16:11:50.204115 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.204082    4549 scope.go:122] "RemoveContainer" containerID="704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"
Oct 31 16:11:50.214864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.214847    4549 scope.go:122] "RemoveContainer" containerID="704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"
Oct 31 16:11:50.215121 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:50.215102    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f\": container with ID starting with 704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f not found: ID does not exist" containerID="704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"
Oct 31 16:11:50.215195 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.215129    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f"} err="failed to get container status \"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f\": rpc error: code = NotFound desc = could not find container \"704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f\": container with ID starting with 704c86c70809e8c9871a7d1f7373d7d82b0da0d8e80bb457fc0a4c6b833dec4f not found: ID does not exist"
Oct 31 16:11:50.366787 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.366712    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 16:11:50.370315 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.370291    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-dff98979c-sr98m"]
Oct 31 16:11:50.608153 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.608123    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:50.691763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691694    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691743    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691766    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691817    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691836    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691860    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691895    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691974    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.691993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.691980    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:50.692370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692003    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\") pod \"3c1c0578-eb92-4770-b75e-be36446122a8\" (UID: \"3c1c0578-eb92-4770-b75e-be36446122a8\") "
Oct 31 16:11:50.692370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692170    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:50.692370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692187    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:50.692370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692273    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:11:50.692370 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692280    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:50.692662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692423    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.692662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692438    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c1c0578-eb92-4770-b75e-be36446122a8-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.692662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692458    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.692662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692474    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.692662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.692489    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.694113 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.694094    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:50.694189 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.694172    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:11:50.694825 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.694805    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:50.694825 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.694813    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm" pod "3c1c0578-eb92-4770-b75e-be36446122a8" (UID: "3c1c0578-eb92-4770-b75e-be36446122a8"). InnerVolumeSpecName "kube-api-access-574mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:50.793090 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.793062    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.793090 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.793087    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-574mm\" (UniqueName: \"kubernetes.io/projected/3c1c0578-eb92-4770-b75e-be36446122a8-kube-api-access-574mm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.793090 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.793097    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c1c0578-eb92-4770-b75e-be36446122a8-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:50.793541 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:50.793112    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c1c0578-eb92-4770-b75e-be36446122a8-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:51.211142 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.211102    4549 generic.go:356] "Generic (PLEG): container finished" podID="3c1c0578-eb92-4770-b75e-be36446122a8" containerID="eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b" exitCode=0
Oct 31 16:11:51.211337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.211157    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"
Oct 31 16:11:51.211337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.211187    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" event={"ID":"3c1c0578-eb92-4770-b75e-be36446122a8","Type":"ContainerDied","Data":"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"}
Oct 31 16:11:51.211337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.211245    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh" event={"ID":"3c1c0578-eb92-4770-b75e-be36446122a8","Type":"ContainerDied","Data":"a12b063ce56c133400ce09c798f2f10392f221b0a14dc7fc2f090e415ab19e2c"}
Oct 31 16:11:51.211337 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.211270    4549 scope.go:122] "RemoveContainer" containerID="eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"
Oct 31 16:11:51.220932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.220914    4549 scope.go:122] "RemoveContainer" containerID="eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"
Oct 31 16:11:51.221205 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:51.221187    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b\": container with ID starting with eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b not found: ID does not exist" containerID="eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"
Oct 31 16:11:51.221255 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.221215    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b"} err="failed to get container status \"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b\": rpc error: code = NotFound desc = could not find container \"eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b\": container with ID starting with eb16714e5a9f437c964cbf016e305f375f1a1887ad4354b60fc02404bad0827b not found: ID does not exist"
Oct 31 16:11:51.378060 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.378022    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:51.383432 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.383405    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bkjsh"]
Oct 31 16:11:51.417765 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.417733    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" path="/var/lib/kubelet/pods/3c1c0578-eb92-4770-b75e-be36446122a8/volumes"
Oct 31 16:11:51.418106 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.418094    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f43ebaf-2e1d-4f28-9e85-71f708747990" path="/var/lib/kubelet/pods/4f43ebaf-2e1d-4f28-9e85-71f708747990/volumes"
Oct 31 16:11:51.941874 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.941848    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:51.944436 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:51.944421    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:52.003882 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003852    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.003882 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003884    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003904    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003920    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003963    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.003981    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004025    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004070    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004097    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004147 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004124    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004155    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004194    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004205    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004237    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004308    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004339    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004354    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004426    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004464    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\") pod \"4953c2db-379c-410d-aea7-732463aa4481\" (UID: \"4953c2db-379c-410d-aea7-732463aa4481\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004503    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004537    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\") pod \"b590ab7c-1c17-447c-adb2-7dab068f729a\" (UID: \"b590ab7c-1c17-447c-adb2-7dab068f729a\") "
Oct 31 16:11:52.004612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004565    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004682    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004895    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004926    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.004964    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005135    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005248    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005429    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005449    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005461    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005474    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005486    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005497    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b590ab7c-1c17-447c-adb2-7dab068f729a-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005523    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005536    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005548    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.006100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.005560    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4953c2db-379c-410d-aea7-732463aa4481-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.007935 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.007895    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:11:52.007935 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.007908    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:52.008088 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.007971    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.008088 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.007974    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:11:52.008240 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.008221    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:52.008309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.008252    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:11:52.008309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.008275    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm" pod "4953c2db-379c-410d-aea7-732463aa4481" (UID: "4953c2db-379c-410d-aea7-732463aa4481"). InnerVolumeSpecName "kube-api-access-tl8sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:52.008773 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.008754    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57" pod "b590ab7c-1c17-447c-adb2-7dab068f729a" (UID: "b590ab7c-1c17-447c-adb2-7dab068f729a"). InnerVolumeSpecName "kube-api-access-29l57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106854    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4953c2db-379c-410d-aea7-732463aa4481-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106882    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106891    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106900    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4953c2db-379c-410d-aea7-732463aa4481-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106910    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8sm\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-kube-api-access-tl8sm\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106919    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106928    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4953c2db-379c-410d-aea7-732463aa4481-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.106932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.106936    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29l57\" (UniqueName: \"kubernetes.io/projected/b590ab7c-1c17-447c-adb2-7dab068f729a-kube-api-access-29l57\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:11:52.219844 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.219813    4549 generic.go:356] "Generic (PLEG): container finished" podID="4953c2db-379c-410d-aea7-732463aa4481" containerID="1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6" exitCode=0
Oct 31 16:11:52.220033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.219858    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"
Oct 31 16:11:52.220033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.219891    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" event={"ID":"4953c2db-379c-410d-aea7-732463aa4481","Type":"ContainerDied","Data":"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"}
Oct 31 16:11:52.220033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.219944    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9" event={"ID":"4953c2db-379c-410d-aea7-732463aa4481","Type":"ContainerDied","Data":"7778abe34cd1aab905ad110b76685fa11616b7b26be55b295929bfab00c2151b"}
Oct 31 16:11:52.220033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.219975    4549 scope.go:122] "RemoveContainer" containerID="1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"
Oct 31 16:11:52.222654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.222627    4549 generic.go:356] "Generic (PLEG): container finished" podID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerID="fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb" exitCode=0
Oct 31 16:11:52.222778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.222682    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" event={"ID":"b590ab7c-1c17-447c-adb2-7dab068f729a","Type":"ContainerDied","Data":"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"}
Oct 31 16:11:52.222778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.222691    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"
Oct 31 16:11:52.222778 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.222718    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k" event={"ID":"b590ab7c-1c17-447c-adb2-7dab068f729a","Type":"ContainerDied","Data":"f18712af7615182f7259365ecee6294b193f5724b5bb918e406e224824a38445"}
Oct 31 16:11:52.229650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.229633    4549 scope.go:122] "RemoveContainer" containerID="1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"
Oct 31 16:11:52.229910 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:52.229892    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6\": container with ID starting with 1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6 not found: ID does not exist" containerID="1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"
Oct 31 16:11:52.229961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.229918    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6"} err="failed to get container status \"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6\": rpc error: code = NotFound desc = could not find container \"1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6\": container with ID starting with 1bc48ad9e18d8cdc9a1dc42ca6675417c2de6c839b406462deabedbcc8a340e6 not found: ID does not exist"
Oct 31 16:11:52.229961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.229934    4549 scope.go:122] "RemoveContainer" containerID="fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"
Oct 31 16:11:52.244480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.244455    4549 scope.go:122] "RemoveContainer" containerID="fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"
Oct 31 16:11:52.244798 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:11:52.244780    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb\": container with ID starting with fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb not found: ID does not exist" containerID="fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"
Oct 31 16:11:52.244862 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.244806    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb"} err="failed to get container status \"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb\": rpc error: code = NotFound desc = could not find container \"fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb\": container with ID starting with fb085cdc05b0a94b3a4274e0127db90427d01fa252a40aed8bb9b136537aefdb not found: ID does not exist"
Oct 31 16:11:52.381990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.381900    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:52.385977 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.385954    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-bpdg9"]
Oct 31 16:11:52.493792 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.493754    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:52.500932 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:52.500905    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-c772k"]
Oct 31 16:11:53.417915 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:53.417880    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4953c2db-379c-410d-aea7-732463aa4481" path="/var/lib/kubelet/pods/4953c2db-379c-410d-aea7-732463aa4481/volumes"
Oct 31 16:11:53.418471 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:11:53.418446    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" path="/var/lib/kubelet/pods/b590ab7c-1c17-447c-adb2-7dab068f729a/volumes"
Oct 31 16:12:07.011373 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:12:07.011332    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:12:08.910548 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910516    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"]
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910865    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4f43ebaf-2e1d-4f28-9e85-71f708747990" containerName="manager"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910877    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f43ebaf-2e1d-4f28-9e85-71f708747990" containerName="manager"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910901    4549 cpu_manager.go:414] "containerMap: removing container" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910906    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910913    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910918    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910931    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy"
Oct 31 16:12:08.910956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910936    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy"
Oct 31 16:12:08.911277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.910990    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c0578-eb92-4770-b75e-be36446122a8" containerName="istio-proxy"
Oct 31 16:12:08.911277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.911001    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f43ebaf-2e1d-4f28-9e85-71f708747990" containerName="manager"
Oct 31 16:12:08.911277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.911010    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="4953c2db-379c-410d-aea7-732463aa4481" containerName="istio-proxy"
Oct 31 16:12:08.911277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.911015    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b590ab7c-1c17-447c-adb2-7dab068f729a" containerName="istio-proxy"
Oct 31 16:12:08.915735 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.915716    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:08.971633 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:08.971598    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\""
Oct 31 16:12:09.021086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.021052    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"]
Oct 31 16:12:09.053702 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053667    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.053702 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053703    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.053905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053725    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.053905 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053835    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.053982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053936    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.054018 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.053982    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.054018 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.054004    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.054087 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.054050    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.054119 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.054087    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82rm\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-kube-api-access-s82rm\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155012 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.154975    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155012 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155015    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155040    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155069    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155102    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155118    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155133    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155257 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155172    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82rm\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-kube-api-access-s82rm\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155274    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155458    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155481    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155538    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155880 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155776    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.155985 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.155963    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.157829 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.157805    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.158026 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.158008    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.211429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.211371    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.211560 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.211546    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82rm\" (UniqueName: \"kubernetes.io/projected/9bc6f8c7-eb08-4416-9cc7-18b828abcca1-kube-api-access-s82rm\") pod \"maas-default-gateway-openshift-default-67bf986f99-hrqdl\" (UID: \"9bc6f8c7-eb08-4416-9cc7-18b828abcca1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.228055 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.228034    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:09.648144 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.648121    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"]
Oct 31 16:12:09.650337 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:12:09.650309    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc6f8c7_eb08_4416_9cc7_18b828abcca1.slice/crio-7b3778f6c62ce6489d963a836b7055db9079197cf30b162a49d6c1c89027bc2c WatchSource:0}: Error finding container 7b3778f6c62ce6489d963a836b7055db9079197cf30b162a49d6c1c89027bc2c: Status 404 returned error can't find the container with id 7b3778f6c62ce6489d963a836b7055db9079197cf30b162a49d6c1c89027bc2c
Oct 31 16:12:09.652408 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.652377    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:12:09.652494 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.652438    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:12:09.652494 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:09.652467    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:12:10.336033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:10.335998    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" event={"ID":"9bc6f8c7-eb08-4416-9cc7-18b828abcca1","Type":"ContainerStarted","Data":"01729034a256858b282eba818e38766244e803dfd9a64536b270ac15c0faa514"}
Oct 31 16:12:10.336443 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:10.336055    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" event={"ID":"9bc6f8c7-eb08-4416-9cc7-18b828abcca1","Type":"ContainerStarted","Data":"7b3778f6c62ce6489d963a836b7055db9079197cf30b162a49d6c1c89027bc2c"}
Oct 31 16:12:10.502648 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:10.502403    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podStartSLOduration=2.502382839 podStartE2EDuration="2.502382839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:10.499996884 +0000 UTC m=+5171.226277862" watchObservedRunningTime="2026-10-31 16:12:10.502382839 +0000 UTC m=+5171.228663818"
Oct 31 16:12:11.228648 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:11.228608    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:11.229971 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:11.229944    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:11.230106 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:11.230008    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:12.229228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:12.229181    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:12.229771 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:12.229246    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:13.229140 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:13.229108    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:13.229341 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:13.229163    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:14.228784 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:14.228751    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:14.228972 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:14.228807    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:15.229397 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:15.229356    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:15.229836 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:15.229416    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:16.228600 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:16.228545    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:16.228769 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:16.228629    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:17.228413 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:17.228379    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 31 16:12:17.228857 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:17.228436    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 31 16:12:18.774454 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:18.774412    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:18.774957 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:18.774766    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:18.775688 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:12:18.775668    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl"
Oct 31 16:12:37.010596 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:12:37.010534    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:13:07.011065 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:07.011027    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:13:09.403404 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.403357    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"]
Oct 31 16:13:09.407258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.407233    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.507080 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.507048    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"]
Oct 31 16:13:09.535350 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535321    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6xg\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-kube-api-access-rn6xg\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535537 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535377    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535537 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535404    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535537 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535520    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535555    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535666    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535688    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535713    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.535899 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.535730    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636412 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636384    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636427    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636491    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636520    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636553    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.636634 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636600    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.637081 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636639    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6xg\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-kube-api-access-rn6xg\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.637081 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636871    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.637081 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.636907    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.637081 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.637027    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.637680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.637655    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.638216 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.638096    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.638216 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.638141    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.638623 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.638599    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.639379 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.639356    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.639702 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.639678    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.700117 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.700035    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.700276 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.700153    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6xg\" (UniqueName: \"kubernetes.io/projected/d2a18e51-b0aa-4cf8-ae11-961e6f757e1b-kube-api-access-rn6xg\") pod \"maas-default-gateway-openshift-default-67bf986f99-szjt6\" (UID: \"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:09.720948 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:09.720922    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:10.148602 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.148556    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"]
Oct 31 16:13:10.150716 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:13:10.150689    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a18e51_b0aa_4cf8_ae11_961e6f757e1b.slice/crio-8beb24aaf2db3fefa7e87031982c338887ef9ad48380041d3cf36503eede06d1 WatchSource:0}: Error finding container 8beb24aaf2db3fefa7e87031982c338887ef9ad48380041d3cf36503eede06d1: Status 404 returned error can't find the container with id 8beb24aaf2db3fefa7e87031982c338887ef9ad48380041d3cf36503eede06d1
Oct 31 16:13:10.152897 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.152832    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:10.153004 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.152964    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:10.153067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.153013    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:10.646499 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.646468    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" event={"ID":"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b","Type":"ContainerStarted","Data":"24cd1da341410892216c8937423d9da3d8f1027c52705418cfb1b8a33cf42677"}
Oct 31 16:13:10.647033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.646518    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" event={"ID":"d2a18e51-b0aa-4cf8-ae11-961e6f757e1b","Type":"ContainerStarted","Data":"8beb24aaf2db3fefa7e87031982c338887ef9ad48380041d3cf36503eede06d1"}
Oct 31 16:13:10.722749 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.722702    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:10.761629 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:10.761549    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podStartSLOduration=1.761533788 podStartE2EDuration="1.761533788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:10.76055269 +0000 UTC m=+5231.486833678" watchObservedRunningTime="2026-10-31 16:13:10.761533788 +0000 UTC m=+5231.487814765"
Oct 31 16:13:11.724395 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:11.724361    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:11.724811 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:11.724412    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:12.723050 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:12.722996    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:12.723280 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:12.723070    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:13.723015 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:13.722972    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:13.723543 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:13.723027    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:14.723002 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:14.722952    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:14.723202 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:14.723015    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:15.721780 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:15.721744    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:15.722096 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:15.722065    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:16.721946 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:16.721905    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:16.722365 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:16.721979    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:17.721992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:17.721950    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Oct 31 16:13:17.722522 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:17.722011    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Oct 31 16:13:19.722056 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:19.722012    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:19.722496 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:19.722086    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:20.762774 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:20.762733    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:20.763217 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:20.762806    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:21.803784 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:21.803736    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:21.804319 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:21.803831    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:22.804623 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:22.804555    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:22.805131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:22.805096    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:23.737664 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.737281    4549 generic.go:356] "Generic (PLEG): container finished" podID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6" exitCode=1
Oct 31 16:13:23.737664 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.737420    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" event={"ID":"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7","Type":"ContainerDied","Data":"5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"}
Oct 31 16:13:23.738981 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.738262    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:23.743729 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.742495    4549 generic.go:356] "Generic (PLEG): container finished" podID="206ba35a-15a9-46f1-b437-582487cb1b8d" containerID="68d49a625a255be09b6198376e41174b026dd6d63941dcce1bf1ca87dcabf1f7" exitCode=1
Oct 31 16:13:23.743729 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.742556    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c" event={"ID":"206ba35a-15a9-46f1-b437-582487cb1b8d","Type":"ContainerDied","Data":"68d49a625a255be09b6198376e41174b026dd6d63941dcce1bf1ca87dcabf1f7"}
Oct 31 16:13:23.743729 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.742930    4549 scope.go:122] "RemoveContainer" containerID="68d49a625a255be09b6198376e41174b026dd6d63941dcce1bf1ca87dcabf1f7"
Oct 31 16:13:23.848667 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.847707    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:23.848667 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:23.847772    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:24.721603 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:24.721527    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:24.741512 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:24.741475    4549 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError"
Oct 31 16:13:24.741860 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:24.741823    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:24.755266 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:24.755238    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c" event={"ID":"206ba35a-15a9-46f1-b437-582487cb1b8d","Type":"ContainerStarted","Data":"5e997eed9904387d815a122a96d8f6843f9c997ab0aa9504ecbfa516f6aac20c"}
Oct 31 16:13:24.755780 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:24.755757    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 16:13:24.889718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:24.889682    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:24.890112 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:24.889741    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:25.436670 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:25.436634    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:25.437756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:25.437732    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6"
Oct 31 16:13:26.150173 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:26.150127    4549 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-115.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:27.286982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:27.286944    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 16:13:27.287409 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:27.287297    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:27.287523 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:27.287503    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7)\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:34.816785 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:34.816734    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:13:34.817333 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:34.816803    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:35.412614 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:35.412563    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:35.765639 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:35.765538    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5f948884b-n2v4c"
Oct 31 16:13:36.151026 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:36.150935    4549 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-115.us-west-1.compute.internal?timeout=10s\": context deadline exceeded"
Oct 31 16:13:36.414499 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:36.414424    4549 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError"
Oct 31 16:13:36.414499 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:36.414464    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:37.017038 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:37.016999    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:13:42.423987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:42.423908    4549 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 16:13:42.424460 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:42.424209    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:42.424460 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:42.424415    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7)\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:44.412701 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:44.412671    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:45.421891 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:45.421847    4549 kuberuntime_manager.go:1707] "Unhandled Error" err="container manager start failed in pod dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7): CreateContainerConfigError: failed to sync configmap cache: timed out waiting for the condition" logger="UnhandledError"
Oct 31 16:13:45.422413 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:45.422082    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CreateContainerConfigError: \"failed to sync configmap cache: timed out waiting for the condition\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:46.151395 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:46.151348    4549 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-115.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:56.151835 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:56.151782    4549 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-115.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.957727    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.957756    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.957788    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.957759    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958042    4549 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958088    4549 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958129    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958174    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958210    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958245    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958279    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958314    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958348    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958383    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958418    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958453    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958487    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-dockercfg-jngf7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958537    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958597    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4b9fp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958634    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958681    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.958763 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958719    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958912    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958959    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.958997    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959036    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-76klk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959076    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959117    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959155    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959193    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959232    4549 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959270    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959311    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959349    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959390    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959494    4549 trace.go:236] Trace[1623163164]: "Reflector WatchList" name:object-"kuadrant-system"/"dns-operator-controller-env" (31-Oct-2026 16:13:23.740) (total time: 33218ms):
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: Trace[1623163164]: [33.218735797s] [33.218735797s] END
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.959960    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960023    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960067    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.960094 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960099    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960131    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960160    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960185    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960216    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:56.960117    4549 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/openshift-ingress/events/maas-default-gateway-openshift-default-67bf986f99-szjt6.18e3a9454c787da3\": http2: client connection lost" event=<
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]:         &Event{ObjectMeta:{maas-default-gateway-openshift-default-67bf986f99-szjt6.18e3a9454c787da3  openshift-ingress   151415 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-ingress,Name:maas-default-gateway-openshift-default-67bf986f99-szjt6,UID:d2a18e51-b0aa-4cf8-ae11-961e6f757e1b,APIVersion:v1,ResourceVersion:151350,FieldPath:spec.containers{istio-proxy},},Reason:ProbeError,Message:Startup probe error: Get "http://10.134.0.198:15021/healthz/ready": dial tcp 10.134.0.198:15021: connect: connection refused
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]:         body:
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]:         ,Source:EventSource{Component:kubelet,Host:ip-10-0-139-115.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 16:13:11 +0000 UTC,LastTimestamp:2026-10-31 16:13:12.723042689 +0000 UTC m=+5233.449323668,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-115.us-west-1.compute.internal,}
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]:  >
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960245    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-72qw8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960287    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960286    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960315    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sn8hj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960336    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960343    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960372    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960382    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960400    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960426    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960437    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960461    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960469    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960509    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960519    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960546    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-glp2g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960602    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960620    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960635    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960654    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960670    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4jx6s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960686    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960698    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960720    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960736    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960751    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960769    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960775    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960792    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960815    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960839    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960859    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960813    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960876    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960893    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960918    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960943    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960942    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960842    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960967    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960916    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960990    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.960999    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961051    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961095    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961116    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961094    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961121    4549 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961141    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961152    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961176    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j72m7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961179    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c265q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961199    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961056    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961222    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-arg423tua776m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961237    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961255    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961280    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961281    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-px9bk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961303    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l5cfh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961253    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961325    4549 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.961309 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961327    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961374    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961403    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961438    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g2n6k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961466    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961500    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961534    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961548    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961597    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961600    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961638    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961641    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961659    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961678    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961700    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjrh5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961714    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961755    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961770    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961777    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961817    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:56.961834    4549 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-115.us-west-1.compute.internal?timeout=10s\": http2: client connection lost"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961859    4549 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961855    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j8ksf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961917    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961943    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961968    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.961992    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gfp4g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962016    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962052    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962085    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xhl5g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962108    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962111    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962135    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962167    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962179    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962195    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:56.964099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:56.962227    4549 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 16:13:57.117730 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.117643    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 16:13:57.815424 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.815391    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\""
Oct 31 16:13:57.828720 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.828697    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 16:13:57.964824 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.964794    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 16:13:57.964824 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.964812    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 16:13:57.965056 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.964872    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 16:13:57.977900 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.977875    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 16:13:57.978068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.977920    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 16:13:57.978068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.977958    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 16:13:57.978068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.978004    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 16:13:57.978068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:57.978048    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gfp4g\""
Oct 31 16:13:58.014524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014481    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.014524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014503    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.014524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014482    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.014827 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014583    4549 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:13:58.014827 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014633    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.014827 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.014762    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 16:13:58.027776 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027755    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 16:13:58.027776 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027759    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j8ksf\""
Oct 31 16:13:58.027973 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027789    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 16:13:58.027973 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027838    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l5cfh\""
Oct 31 16:13:58.027973 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027956    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.027973 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027959    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 16:13:58.028138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.027991    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 16:13:58.028138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.028001    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 16:13:58.028138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.028030    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.028138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.028096    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 16:13:58.065154 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.065131    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 16:13:58.065307 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.065238    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\""
Oct 31 16:13:58.078389 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.078288    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 16:13:58.078617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.078386    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 16:13:58.078732 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.078617    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.078732 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.078691    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\""
Oct 31 16:13:58.078915 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.078896    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 16:13:58.079493 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.079473    4549 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 16:13:58.085286 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.085266    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"]
Oct 31 16:13:58.089663 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.089643    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.114498 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114469    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\""
Oct 31 16:13:58.114626 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114538    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-px9bk\""
Oct 31 16:13:58.114718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114699    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114705    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114726    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114708    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114755    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114762    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\""
Oct 31 16:13:58.114786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114751    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 16:13:58.115107 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.114819    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\""
Oct 31 16:13:58.115336 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.115320    4549 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:13:58.121930 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.121910    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 16:13:58.125826 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.125807    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 16:13:58.128325 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.128306    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.129208 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.129191    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\""
Oct 31 16:13:58.129485 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.129470    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\""
Oct 31 16:13:58.130440 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.130426    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 16:13:58.133063 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.133050    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.133659 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.133644    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j72m7\""
Oct 31 16:13:58.135650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.135632    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.136539 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.136521    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c265q\""
Oct 31 16:13:58.144382 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.144354    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\""
Oct 31 16:13:58.146934 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.146913    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.150893 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.150872    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xhl5g\""
Oct 31 16:13:58.152950 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.152930    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 16:13:58.158908 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.158892    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 16:13:58.158988 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.158948    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 16:13:58.174700 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.174679    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 16:13:58.183580 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.183558    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 16:13:58.189423 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.189396    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 16:13:58.189513 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.189429    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 16:13:58.196768 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.196744    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 16:13:58.196902 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.196888    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 16:13:58.197016 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.196997    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 16:13:58.197205 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.197185    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\""
Oct 31 16:13:58.201237 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201214    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201253    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4kq\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-kube-api-access-xk4kq\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201275    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201329    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201513 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201428    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201513 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201465    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201620 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201556    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201676 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201625    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.201676 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.201650    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.206537 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.206513    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.213653 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.213618    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.215678 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.215664    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\""
Oct 31 16:13:58.220834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.220819    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.227369 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.227351    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 16:13:58.240931 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.240910    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 16:13:58.241025 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.240945    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 16:13:58.247449 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.247429    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.247540 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.247486    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 16:13:58.253555 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.253542    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 16:13:58.275518 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.275496    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 16:13:58.281278 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.281261    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 16:13:58.288310 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.288293    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.302218 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302195    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302316 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302228    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302316 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302256    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302316 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302286    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4kq\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-kube-api-access-xk4kq\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302316 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302307    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302332    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302365    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302383    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302437    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302734    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302754    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.302853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302813    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.303033 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.302872    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.303224 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.303203    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.304826 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.304807    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.305110 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.305094    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.310317 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.310299    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.311807 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.311791    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.321194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.321179    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 16:13:58.322717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.322703    4549 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:13:58.331993 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.331951    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-76klk\""
Oct 31 16:13:58.335127 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.335109    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\""
Oct 31 16:13:58.341079 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.341067    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 16:13:58.341079 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.341073    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 16:13:58.344685 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.344669    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.346340 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.346325    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 16:13:58.346340 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.346331    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-4b9fp\""
Oct 31 16:13:58.357720 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.357703    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-sn8hj\""
Oct 31 16:13:58.359626 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.359565    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.359748 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.359732    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4kq\" (UniqueName: \"kubernetes.io/projected/4e7d9582-0fa5-452b-8160-297ef97ec4d2-kube-api-access-xk4kq\") pod \"maas-default-gateway-openshift-default-67bf986f99-nzdpc\" (UID: \"4e7d9582-0fa5-452b-8160-297ef97ec4d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.370477 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.370461    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.370606 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.370524    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.380341 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.380320    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 16:13:58.381494 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.381476    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\""
Oct 31 16:13:58.386211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.386196    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.393777 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.393758    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.403122 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.403099    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:58.411986 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.411968    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:13:58.412282 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:13:58.412256    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-8557cbdb84-52jvs_kuadrant-system(fa6cac91-ffce-4ec7-8c7d-1fc35213fef7)\"" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" podUID="fa6cac91-ffce-4ec7-8c7d-1fc35213fef7"
Oct 31 16:13:58.416044 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.416021    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 16:13:58.416771 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.416752    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jngf7\""
Oct 31 16:13:58.418922 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.418902    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 16:13:58.421422 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.421401    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 16:13:58.421533 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.421512    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\""
Oct 31 16:13:58.422511 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.422490    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 16:13:58.433830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.433808    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.438259 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.438228    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 16:13:58.463116 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.461400    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.467285 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.467232    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.473609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.469107    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 16:13:58.473609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.469260    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 16:13:58.475717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.475681    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjrh5\""
Oct 31 16:13:58.481012 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.480986    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.487519 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.487492    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\""
Oct 31 16:13:58.495259 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.490593    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 16:13:58.495259 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.495133    4549 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 16:13:58.504483 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.504362    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4jx6s\""
Oct 31 16:13:58.507333 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.507310    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-glp2g\""
Oct 31 16:13:58.515417 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.515397    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 16:13:58.519840 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.519793    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 16:13:58.533617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.531299    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-arg423tua776m\""
Oct 31 16:13:58.545782 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.545759    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\""
Oct 31 16:13:58.554806 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.554788    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.559711 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.559693    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-72qw8\""
Oct 31 16:13:58.559943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.559926    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 16:13:58.560369 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.560354    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\""
Oct 31 16:13:58.561712 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.561696    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 16:13:58.573002 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.572979    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.575024 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.575006    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\""
Oct 31 16:13:58.575143 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.575132    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 16:13:58.575341 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.575328    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 16:13:58.576881 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.576867    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.577467 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.577454    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 16:13:58.581750 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.581736    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 16:13:58.584718 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.584675    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g2n6k\""
Oct 31 16:13:58.594478 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.594460    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 16:13:58.600909 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.600893    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 16:13:58.605338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.605324    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 16:13:58.605723 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.605708    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 16:13:58.834682 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.834654    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"]
Oct 31 16:13:58.836868 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:13:58.836811    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7d9582_0fa5_452b_8160_297ef97ec4d2.slice/crio-79136446bf2d28f6d896a9b86b641a82f36925979ededd07e2ebc19de27380f0 WatchSource:0}: Error finding container 79136446bf2d28f6d896a9b86b641a82f36925979ededd07e2ebc19de27380f0: Status 404 returned error can't find the container with id 79136446bf2d28f6d896a9b86b641a82f36925979ededd07e2ebc19de27380f0
Oct 31 16:13:58.839076 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.839042    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:58.839141 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.839111    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:58.839178 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.839151    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:13:58.952077 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.952048    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" event={"ID":"4e7d9582-0fa5-452b-8160-297ef97ec4d2","Type":"ContainerStarted","Data":"1ae5ff4991ef04385fedc6646b6d0f891c3af311af7f90bba8d6f1159890c041"}
Oct 31 16:13:58.952242 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:58.952097    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" event={"ID":"4e7d9582-0fa5-452b-8160-297ef97ec4d2","Type":"ContainerStarted","Data":"79136446bf2d28f6d896a9b86b641a82f36925979ededd07e2ebc19de27380f0"}
Oct 31 16:13:59.403497 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:59.403468    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:13:59.404858 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:59.404831    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:13:59.404992 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:13:59.404883    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:00.408040 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:00.407964    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:00.963626 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:00.963564    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:00.964734 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:00.964719    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:03.022717 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:03.022647    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podStartSLOduration=39.022623825 podStartE2EDuration="39.022623825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:14:03.012747354 +0000 UTC m=+5283.739028331" watchObservedRunningTime="2026-10-31 16:14:03.022623825 +0000 UTC m=+5283.748904805"
Oct 31 16:14:06.412783 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.412753    4549 scope.go:122] "RemoveContainer" containerID="5f0eeba93bff8e11b24c4199968ae8861335ed77ec50260457aaf6e7a16c56c6"
Oct 31 16:14:06.728472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.728380    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 16:14:06.728472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.728422    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 16:14:06.728472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.728432    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 16:14:06.728472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.728445    4549 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-115.us-west-1.compute.internal" event="NodeReady"
Oct 31 16:14:06.999336 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.999248    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs" event={"ID":"fa6cac91-ffce-4ec7-8c7d-1fc35213fef7","Type":"ContainerStarted","Data":"5cc5dda9cfb56a2b3a9e252b6aecae34ceea3425b14d5446fe132cfb4f936f81"}
Oct 31 16:14:06.999522 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:06.999477    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 16:14:07.011384 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:14:07.011349    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:14:18.006501 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:18.006469    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-52jvs"
Oct 31 16:14:21.082218 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.082181    4549 generic.go:356] "Generic (PLEG): container finished" podID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerID="1ae5ff4991ef04385fedc6646b6d0f891c3af311af7f90bba8d6f1159890c041" exitCode=0
Oct 31 16:14:21.082647 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.082257    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" event={"ID":"4e7d9582-0fa5-452b-8160-297ef97ec4d2","Type":"ContainerDied","Data":"1ae5ff4991ef04385fedc6646b6d0f891c3af311af7f90bba8d6f1159890c041"}
Oct 31 16:14:21.082765 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.082749    4549 scope.go:122] "RemoveContainer" containerID="1ae5ff4991ef04385fedc6646b6d0f891c3af311af7f90bba8d6f1159890c041"
Oct 31 16:14:21.084086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.084058    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:14:21.084211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.084139    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:14:21.084211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:21.084189    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:14:22.090398 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:22.090360    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" event={"ID":"4e7d9582-0fa5-452b-8160-297ef97ec4d2","Type":"ContainerStarted","Data":"63731d056583d2aedc4c0f0512b828ac145793437eb065d19d5ffcaaf5009f36"}
Oct 31 16:14:22.412091 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:22.411997    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:22.412609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:22.412465    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:22.412818 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:22.412795    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:23.480620 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:23.472050    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:23.480620 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:23.472143    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:24.409597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:24.404987    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:24.409597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:24.405053    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:25.410596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:25.405951    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:25.410596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:25.406014    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:26.419603 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:26.412001    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:26.419603 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:26.412061    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:27.404234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:27.404187    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:27.404463 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:27.404259    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:28.404160 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:28.404123    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 31 16:14:28.404564 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:28.404194    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 31 16:14:30.405017 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:30.404974    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:14:30.405511 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:30.405044    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:14:30.963930 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:30.963898    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:31.406276 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:31.406234    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:14:31.406828 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:31.406303    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:14:32.447849 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:32.447797    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:14:32.448525 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:32.448492    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:14:33.490931 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:33.490895    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:14:33.491446 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:33.490954    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:14:34.065025 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:34.064981    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:34.066883 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:34.066857    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc"
Oct 31 16:14:34.820832 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:34.820796    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:14:34.821486 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:14:34.821437    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:14:37.017957 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:14:37.017913    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:15:00.152172 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.152133    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:00.155659 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.155642    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.206234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.206194    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.206415 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.206244    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngzv\" (UniqueName: \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.219805 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.219768    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:00.257705 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.257670    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\""
Oct 31 16:15:00.257705 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.257666    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:15:00.257705 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.257666    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:15:00.306947 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.306905    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngzv\" (UniqueName: \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.307154 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.307021    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.307649 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.307631    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.363592 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.363534    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngzv\" (UniqueName: \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\") pod \"maas-api-key-cleanup-29891055-7ncbr\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.468532 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.468441    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:00.841863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:00.841837    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:00.843510 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:15:00.843478    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac056324_4b1b_4bed_a504_403d484d3344.slice/crio-02105115fda03a12301513fcaca00d053cc557534ea57144c678655f947a12c2 WatchSource:0}: Error finding container 02105115fda03a12301513fcaca00d053cc557534ea57144c678655f947a12c2: Status 404 returned error can't find the container with id 02105115fda03a12301513fcaca00d053cc557534ea57144c678655f947a12c2
Oct 31 16:15:01.365716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:01.365604    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerStarted","Data":"bf89ad8b714b2d8f61d40791ff3af674d6f97a0074c6219a70a29c41b5199d9d"}
Oct 31 16:15:01.365716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:01.365683    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerStarted","Data":"02105115fda03a12301513fcaca00d053cc557534ea57144c678655f947a12c2"}
Oct 31 16:15:01.479501 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:01.479439    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" podStartSLOduration=1.479425269 podStartE2EDuration="1.479425269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:01.476429449 +0000 UTC m=+5342.202710429" watchObservedRunningTime="2026-10-31 16:15:01.479425269 +0000 UTC m=+5342.205706246"
Oct 31 16:15:02.372703 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:02.372670    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac056324-4b1b-4bed-a504-403d484d3344" containerID="bf89ad8b714b2d8f61d40791ff3af674d6f97a0074c6219a70a29c41b5199d9d" exitCode=60
Oct 31 16:15:02.373125 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:02.372722    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerDied","Data":"bf89ad8b714b2d8f61d40791ff3af674d6f97a0074c6219a70a29c41b5199d9d"}
Oct 31 16:15:02.373125 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:02.372972    4549 scope.go:122] "RemoveContainer" containerID="bf89ad8b714b2d8f61d40791ff3af674d6f97a0074c6219a70a29c41b5199d9d"
Oct 31 16:15:03.380278 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:03.380241    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac056324-4b1b-4bed-a504-403d484d3344" containerID="a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086" exitCode=60
Oct 31 16:15:03.380765 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:03.380285    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerDied","Data":"a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086"}
Oct 31 16:15:03.380765 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:03.380343    4549 scope.go:122] "RemoveContainer" containerID="bf89ad8b714b2d8f61d40791ff3af674d6f97a0074c6219a70a29c41b5199d9d"
Oct 31 16:15:03.380765 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:03.380638    4549 scope.go:122] "RemoveContainer" containerID="a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086"
Oct 31 16:15:03.380899 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:15:03.380873    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-7ncbr_odh-ai-gateway-infra(ac056324-4b1b-4bed-a504-403d484d3344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" podUID="ac056324-4b1b-4bed-a504-403d484d3344"
Oct 31 16:15:04.386883 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:04.386855    4549 scope.go:122] "RemoveContainer" containerID="a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086"
Oct 31 16:15:04.387286 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:15:04.387074    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-7ncbr_odh-ai-gateway-infra(ac056324-4b1b-4bed-a504-403d484d3344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" podUID="ac056324-4b1b-4bed-a504-403d484d3344"
Oct 31 16:15:07.012101 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:15:07.012057    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:15:16.413239 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:16.413149    4549 scope.go:122] "RemoveContainer" containerID="a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086"
Oct 31 16:15:17.500609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:17.500500    4549 generic.go:356] "Generic (PLEG): container finished" podID="ac056324-4b1b-4bed-a504-403d484d3344" containerID="11819f22eaddb1ee26cfe3271eb6705754477baf24ffe68b98a5cef8ea9e569b" exitCode=60
Oct 31 16:15:17.500609 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:17.500588    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerDied","Data":"11819f22eaddb1ee26cfe3271eb6705754477baf24ffe68b98a5cef8ea9e569b"}
Oct 31 16:15:17.501101 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:17.500655    4549 scope.go:122] "RemoveContainer" containerID="a5a85db344272a52d39cca3c29e9d640644e6f421ebea3807c86da2fda01f086"
Oct 31 16:15:17.501101 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:17.500988    4549 scope.go:122] "RemoveContainer" containerID="11819f22eaddb1ee26cfe3271eb6705754477baf24ffe68b98a5cef8ea9e569b"
Oct 31 16:15:17.501256 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:15:17.501236    4549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891055-7ncbr_odh-ai-gateway-infra(ac056324-4b1b-4bed-a504-403d484d3344)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" podUID="ac056324-4b1b-4bed-a504-403d484d3344"
Oct 31 16:15:18.621892 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.621854    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:18.766196 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.766155    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:18.841236 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.841197    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\") pod \"ac056324-4b1b-4bed-a504-403d484d3344\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") "
Oct 31 16:15:18.841450 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.841266    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\" (UniqueName: \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\") pod \"ac056324-4b1b-4bed-a504-403d484d3344\" (UID: \"ac056324-4b1b-4bed-a504-403d484d3344\") "
Oct 31 16:15:18.841648 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.841616    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca" pod "ac056324-4b1b-4bed-a504-403d484d3344" (UID: "ac056324-4b1b-4bed-a504-403d484d3344"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:18.843492 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.843469    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv" pod "ac056324-4b1b-4bed-a504-403d484d3344" (UID: "ac056324-4b1b-4bed-a504-403d484d3344"). InnerVolumeSpecName "kube-api-access-wngzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:18.942435 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.942340    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac056324-4b1b-4bed-a504-403d484d3344-maas-api-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:18.942435 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:18.942372    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngzv\" (UniqueName: \"kubernetes.io/projected/ac056324-4b1b-4bed-a504-403d484d3344-kube-api-access-wngzv\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:19.514137 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:19.514105    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"
Oct 31 16:15:19.514342 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:19.514140    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr" event={"ID":"ac056324-4b1b-4bed-a504-403d484d3344","Type":"ContainerDied","Data":"02105115fda03a12301513fcaca00d053cc557534ea57144c678655f947a12c2"}
Oct 31 16:15:19.514342 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:19.514200    4549 scope.go:122] "RemoveContainer" containerID="11819f22eaddb1ee26cfe3271eb6705754477baf24ffe68b98a5cef8ea9e569b"
Oct 31 16:15:19.677294 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:19.677216    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:19.726241 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:19.726204    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891055-7ncbr"]
Oct 31 16:15:21.134441 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134409    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"]
Oct 31 16:15:21.134863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134804    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.134863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134815    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.134863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134827    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.134863 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134832    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.135001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134908    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.135001 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.134922    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:21.137856 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.137840    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.162176 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.162140    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.162176 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.162176    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.162387 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.162302    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkkz\" (UniqueName: \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.239692 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.239655    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:15:21.239692 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.239668    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:15:21.239692 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.239666    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:15:21.239946 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.239818    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\""
Oct 31 16:15:21.242478 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.242459    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"]
Oct 31 16:15:21.263524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.263492    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.263524 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.263524    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.263775 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.263616    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkkz\" (UniqueName: \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.264164 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.264144    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.266140 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.266118    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.319915 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.319883    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkkz\" (UniqueName: \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\") pod \"maas-api-revoke-sub-e5ab86fe-mrc6v\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.418308 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.418225    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac056324-4b1b-4bed-a504-403d484d3344" path="/var/lib/kubelet/pods/ac056324-4b1b-4bed-a504-403d484d3344/volumes"
Oct 31 16:15:21.449958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.449929    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:21.819930 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:21.819897    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"]
Oct 31 16:15:21.821462 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:15:21.821438    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91195bcb_1b77_4bbb_86b1_09de0be1698a.slice/crio-83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c WatchSource:0}: Error finding container 83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c: Status 404 returned error can't find the container with id 83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c
Oct 31 16:15:22.533099 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:22.533057    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v" event={"ID":"91195bcb-1b77-4bbb-86b1-09de0be1698a","Type":"ContainerStarted","Data":"6fb86f99d41b96d627f70df678e81330a23c51bdb8efb965a6b82025edeee354"}
Oct 31 16:15:22.533499 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:22.533117    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v" event={"ID":"91195bcb-1b77-4bbb-86b1-09de0be1698a","Type":"ContainerStarted","Data":"83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c"}
Oct 31 16:15:22.646137 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:22.646078    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v" podStartSLOduration=1.6460630200000002 podStartE2EDuration="1.64606302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:22.644426361 +0000 UTC m=+5363.370707339" watchObservedRunningTime="2026-10-31 16:15:22.64606302 +0000 UTC m=+5363.372344048"
Oct 31 16:15:23.539219 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:23.539185    4549 generic.go:356] "Generic (PLEG): container finished" podID="91195bcb-1b77-4bbb-86b1-09de0be1698a" containerID="6fb86f99d41b96d627f70df678e81330a23c51bdb8efb965a6b82025edeee354" exitCode=0
Oct 31 16:15:23.539673 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:23.539278    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v" event={"ID":"91195bcb-1b77-4bbb-86b1-09de0be1698a","Type":"ContainerDied","Data":"6fb86f99d41b96d627f70df678e81330a23c51bdb8efb965a6b82025edeee354"}
Oct 31 16:15:24.677805 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.677780    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:24.792511 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.792480    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\" (UniqueName: \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\") pod \"91195bcb-1b77-4bbb-86b1-09de0be1698a\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") "
Oct 31 16:15:24.792714 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.792523    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\") pod \"91195bcb-1b77-4bbb-86b1-09de0be1698a\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") "
Oct 31 16:15:24.792714 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.792602    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\") pod \"91195bcb-1b77-4bbb-86b1-09de0be1698a\" (UID: \"91195bcb-1b77-4bbb-86b1-09de0be1698a\") "
Oct 31 16:15:24.793829 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.793786    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca" pod "91195bcb-1b77-4bbb-86b1-09de0be1698a" (UID: "91195bcb-1b77-4bbb-86b1-09de0be1698a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:24.796198 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.796083    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca" pod "91195bcb-1b77-4bbb-86b1-09de0be1698a" (UID: "91195bcb-1b77-4bbb-86b1-09de0be1698a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:15:24.796438 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.796412    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz" pod "91195bcb-1b77-4bbb-86b1-09de0be1698a" (UID: "91195bcb-1b77-4bbb-86b1-09de0be1698a"). InnerVolumeSpecName "kube-api-access-jfkkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:24.894067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.894015    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfkkz\" (UniqueName: \"kubernetes.io/projected/91195bcb-1b77-4bbb-86b1-09de0be1698a-kube-api-access-jfkkz\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.894067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.894061    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-xks-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:24.894067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:24.894071    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91195bcb-1b77-4bbb-86b1-09de0be1698a-maas-api-openshift-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:25.550871 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:25.550837    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"
Oct 31 16:15:25.551067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:25.550864    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v" event={"ID":"91195bcb-1b77-4bbb-86b1-09de0be1698a","Type":"ContainerDied","Data":"83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c"}
Oct 31 16:15:25.551067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:25.550922    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83910e4b7d7b40928b13fe6f510f6e2d48811b7eebcc49a1931d7ef276980f3c"
Oct 31 16:15:37.013635 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:15:37.013593    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:15:39.715321 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715281    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"]
Oct 31 16:15:39.715729 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715718    4549 cpu_manager.go:414] "containerMap: removing container" podUID="91195bcb-1b77-4bbb-86b1-09de0be1698a" containerName="revoke-keys"
Oct 31 16:15:39.715775 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715731    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91195bcb-1b77-4bbb-86b1-09de0be1698a" containerName="revoke-keys"
Oct 31 16:15:39.715775 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715745    4549 cpu_manager.go:414] "containerMap: removing container" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:39.715775 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715750    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:39.715884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715829    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="91195bcb-1b77-4bbb-86b1-09de0be1698a" containerName="revoke-keys"
Oct 31 16:15:39.715884 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.715839    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac056324-4b1b-4bed-a504-403d484d3344" containerName="cleanup"
Oct 31 16:15:39.725741 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.725706    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.775774 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.775742    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"]
Oct 31 16:15:39.821234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.821195    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 16:15:39.821429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.821301    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:15:39.821429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.821364    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\""
Oct 31 16:15:39.827140 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.827122    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:15:39.844680 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.844659    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcmf\" (UniqueName: \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.844745 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.844702    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.844745 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.844736    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.945671 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.945625    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcmf\" (UniqueName: \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.945671 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.945674    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.945961 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.945715    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.946384 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.946362    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:39.948301 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:39.948273    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:40.002686 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:40.002596    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcmf\" (UniqueName: \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\") pod \"maas-api-revoke-sub-f00b18ff-4kslg\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:40.041081 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:40.041034    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:40.415331 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:40.415305    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"]
Oct 31 16:15:40.416427 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:15:40.416399    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124e9b4a_bb54_46b8_8df2_750dd48cc655.slice/crio-f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3 WatchSource:0}: Error finding container f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3: Status 404 returned error can't find the container with id f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3
Oct 31 16:15:40.631332 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:40.631300    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg" event={"ID":"124e9b4a-bb54-46b8-8df2-750dd48cc655","Type":"ContainerStarted","Data":"f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3"}
Oct 31 16:15:41.642453 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:41.642348    4549 generic.go:356] "Generic (PLEG): container finished" podID="124e9b4a-bb54-46b8-8df2-750dd48cc655" containerID="8f62e8220ea6f01898d4ba1a89a8bb72e5bc5ef43be460f26f795eca802dfc20" exitCode=0
Oct 31 16:15:41.642989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:41.642448    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg" event={"ID":"124e9b4a-bb54-46b8-8df2-750dd48cc655","Type":"ContainerDied","Data":"8f62e8220ea6f01898d4ba1a89a8bb72e5bc5ef43be460f26f795eca802dfc20"}
Oct 31 16:15:43.071715 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.071216    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:43.185086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.185049    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\") pod \"124e9b4a-bb54-46b8-8df2-750dd48cc655\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") "
Oct 31 16:15:43.185295 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.185171    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\") pod \"124e9b4a-bb54-46b8-8df2-750dd48cc655\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") "
Oct 31 16:15:43.185295 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.185237    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\" (UniqueName: \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\") pod \"124e9b4a-bb54-46b8-8df2-750dd48cc655\" (UID: \"124e9b4a-bb54-46b8-8df2-750dd48cc655\") "
Oct 31 16:15:43.185439 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.185404    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca" pod "124e9b4a-bb54-46b8-8df2-750dd48cc655" (UID: "124e9b4a-bb54-46b8-8df2-750dd48cc655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:15:43.185610 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.185587    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-openshift-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:43.187764 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.187708    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca" pod "124e9b4a-bb54-46b8-8df2-750dd48cc655" (UID: "124e9b4a-bb54-46b8-8df2-750dd48cc655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:15:43.187915 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.187787    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf" pod "124e9b4a-bb54-46b8-8df2-750dd48cc655" (UID: "124e9b4a-bb54-46b8-8df2-750dd48cc655"). InnerVolumeSpecName "kube-api-access-5mcmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:15:43.286584 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.286525    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/124e9b4a-bb54-46b8-8df2-750dd48cc655-maas-api-xks-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:43.286584 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.286588    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcmf\" (UniqueName: \"kubernetes.io/projected/124e9b4a-bb54-46b8-8df2-750dd48cc655-kube-api-access-5mcmf\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:15:43.679974 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.678664    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg" event={"ID":"124e9b4a-bb54-46b8-8df2-750dd48cc655","Type":"ContainerDied","Data":"f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3"}
Oct 31 16:15:43.679974 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.678764    4549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f42386db659bd71cc0b454a26f091f4de74a97a1fc7e6c9357c5c29500392be3"
Oct 31 16:15:43.679974 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:43.678856    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"
Oct 31 16:15:52.842220 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.842175    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:15:52.842792 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.842626    4549 cpu_manager.go:414] "containerMap: removing container" podUID="124e9b4a-bb54-46b8-8df2-750dd48cc655" containerName="revoke-keys"
Oct 31 16:15:52.842792 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.842640    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124e9b4a-bb54-46b8-8df2-750dd48cc655" containerName="revoke-keys"
Oct 31 16:15:52.842792 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.842726    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="124e9b4a-bb54-46b8-8df2-750dd48cc655" containerName="revoke-keys"
Oct 31 16:15:52.845855 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.845833    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:52.898376 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.898328    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-14a7e5-openshift-default-dockercfg-xt9fh\""
Oct 31 16:15:52.953388 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:52.953351    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:15:53.008199 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008160    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008393 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008209    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008393 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008240    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008393 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008350    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008393 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008371    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008393 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008387    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008638 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008405    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008638 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008469    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.008720 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.008645    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mp6\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.109935 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.109837    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.109935 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.109910    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mp6\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.109962    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.109996    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110055    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110194 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110188    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110456 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110218    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110456 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110243    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.110456 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110272    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.111211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.110967    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.111211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.111023    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.111211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.111159    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.111211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.111191    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.111762 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.111738    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.112696 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.112669    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.113264 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.113243    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.167057 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.167017    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mp6\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.167199 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.167053    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\") pod \"e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") " pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.459410 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.459322    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:53.953954 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.953921    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:15:53.970783 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:15:53.969786    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92663597_71f8_449e_a2cc_24c91c36324d.slice/crio-ca25e8fd2c83c70df79746b22dc53524a445bf92c80e92954bd81ae624471ba8 WatchSource:0}: Error finding container ca25e8fd2c83c70df79746b22dc53524a445bf92c80e92954bd81ae624471ba8: Status 404 returned error can't find the container with id ca25e8fd2c83c70df79746b22dc53524a445bf92c80e92954bd81ae624471ba8
Oct 31 16:15:53.972772 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.972724    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:15:53.972912 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.972862    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:15:53.972987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:53.972911    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:15:54.770598 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:54.769896    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerStarted","Data":"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"}
Oct 31 16:15:54.770598 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:54.769964    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerStarted","Data":"ca25e8fd2c83c70df79746b22dc53524a445bf92c80e92954bd81ae624471ba8"}
Oct 31 16:15:54.994153 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:54.994085    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podStartSLOduration=2.994068126 podStartE2EDuration="2.994068126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:15:54.993898018 +0000 UTC m=+5395.720178998" watchObservedRunningTime="2026-10-31 16:15:54.994068126 +0000 UTC m=+5395.720349180"
Oct 31 16:15:55.466596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:55.466263    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:55.474958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:55.474806    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:55.779643 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:55.779610    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:15:55.780738 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:15:55.780712    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:16:01.560281 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.560245    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:16:01.563989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.563972    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.619848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.619817    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:16:01.665878 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.665840    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-14a7e5\""
Oct 31 16:16:01.666086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.665937    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-91c1a072-dockercfg-m5scv\""
Oct 31 16:16:01.724705 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.724658    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sjfq\" (UniqueName: \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.724883 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.724713    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.825848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.825757    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sjfq\" (UniqueName: \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.825848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.825799    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.826499 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.826479    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:01.882554 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:01.882523    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sjfq\" (UniqueName: \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\") pod \"payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:02.176510 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.176418    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:02.547307 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.547248    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:16:02.548381 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:16:02.548349    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a39c72d_aaf7_4fed_99e7_0ec8a6467139.slice/crio-f84ba4a85f6192221c9a29a57048a4bf0a11988ca253d985dddb210f1a6d7df0 WatchSource:0}: Error finding container f84ba4a85f6192221c9a29a57048a4bf0a11988ca253d985dddb210f1a6d7df0: Status 404 returned error can't find the container with id f84ba4a85f6192221c9a29a57048a4bf0a11988ca253d985dddb210f1a6d7df0
Oct 31 16:16:02.820599 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.820486    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" event={"ID":"6a39c72d-aaf7-4fed-99e7-0ec8a6467139","Type":"ContainerStarted","Data":"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"}
Oct 31 16:16:02.820599 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.820545    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" event={"ID":"6a39c72d-aaf7-4fed-99e7-0ec8a6467139","Type":"ContainerStarted","Data":"f84ba4a85f6192221c9a29a57048a4bf0a11988ca253d985dddb210f1a6d7df0"}
Oct 31 16:16:02.821136 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.820684    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:02.934418 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:02.934363    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" podStartSLOduration=1.934346552 podStartE2EDuration="1.934346552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:02.931699296 +0000 UTC m=+5403.657980275" watchObservedRunningTime="2026-10-31 16:16:02.934346552 +0000 UTC m=+5403.660627531"
Oct 31 16:16:03.320376 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:03.320351    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:16:03.478421 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:03.478394    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:16:04.048053 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.048016    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:16:04.052333 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.052313    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.104640 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.104608    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:16:04.148267 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.148201    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.148630 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.148334    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbr7\" (UniqueName: \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.148630 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.148382    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.148630 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.148510    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.154368 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.154340    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-14a7e5-serving-cert\""
Oct 31 16:16:04.154368 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.154355    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:16:04.154617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.154403    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:16:04.154617 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.154421    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 16:16:04.158131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.158110    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:16:04.249383 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.249339    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.249581 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.249410    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.249581 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.249430    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbr7\" (UniqueName: \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.249581 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.249448    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.249581 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:16:04.249521    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-serving-cert: secret "maas-api-e2e-worker-single-main-14a7e5-serving-cert" not found
Oct 31 16:16:04.249835 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:16:04.249613    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls podName:578093aa-2d02-4c82-8dbe-00f91514907e nodeName:}" failed. No retries permitted until 2026-10-31 16:16:04.749594042 +0000 UTC m=+5405.475875007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls") pod "maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" (UID: "578093aa-2d02-4c82-8dbe-00f91514907e") : secret "maas-api-e2e-worker-single-main-14a7e5-serving-cert" not found
Oct 31 16:16:04.250150 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.250131    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.252277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.252247    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.308611 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.308501    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbr7\" (UniqueName: \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.754109 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.754025    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.756615 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.756590    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:04.966012 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:04.965975    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:05.345952 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:05.345924    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:16:05.347533 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:16:05.347509    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578093aa_2d02_4c82_8dbe_00f91514907e.slice/crio-c478ec89f0f651ee3e9dd3e15a66d74d595069873a1220174d5d6606d2cb9f78 WatchSource:0}: Error finding container c478ec89f0f651ee3e9dd3e15a66d74d595069873a1220174d5d6606d2cb9f78: Status 404 returned error can't find the container with id c478ec89f0f651ee3e9dd3e15a66d74d595069873a1220174d5d6606d2cb9f78
Oct 31 16:16:05.349068 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:05.349048    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:16:05.837790 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:05.837750    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" event={"ID":"578093aa-2d02-4c82-8dbe-00f91514907e","Type":"ContainerStarted","Data":"c478ec89f0f651ee3e9dd3e15a66d74d595069873a1220174d5d6606d2cb9f78"}
Oct 31 16:16:06.846430 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:06.846397    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" event={"ID":"578093aa-2d02-4c82-8dbe-00f91514907e","Type":"ContainerStarted","Data":"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"}
Oct 31 16:16:06.846878 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:06.846528    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:06.960458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:06.960400    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" podStartSLOduration=1.962299354 podStartE2EDuration="2.960384019s" totalImagesPullingTime="998.084665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:16:06.959251923 +0000 UTC m=+5407.685532902" watchObservedRunningTime="2026-10-31 16:16:06.960384019 +0000 UTC m=+5407.686664997"
Oct 31 16:16:07.012034 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:16:07.011999    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:16:12.858612 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:12.858561    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:16:13.829987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:13.829960    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:16:20.952107 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.952077    4549 generic.go:356] "Generic (PLEG): container finished" podID="92663597-71f8-449e-a2cc-24c91c36324d" containerID="b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7" exitCode=0
Oct 31 16:16:20.952483 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.952149    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerDied","Data":"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"}
Oct 31 16:16:20.952549 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.952535    4549 scope.go:122] "RemoveContainer" containerID="b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"
Oct 31 16:16:20.953907 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.953878    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:16:20.953977 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.953951    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:16:20.954020 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:20.953983    4549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 16:16:21.959597 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:21.959539    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerStarted","Data":"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"}
Oct 31 16:16:22.460473 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:22.460443    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:16:22.461648 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:22.461621    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 16:16:22.461787 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:22.461673    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 16:16:23.459881 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:23.459847    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 16:16:23.460297 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:23.459901    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 16:16:24.460221 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:24.460184    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 16:16:24.460681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:24.460242    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 16:16:25.460139 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:25.460103    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 16:16:25.460318 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:25.460161    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 16:16:25.779985 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:25.779957    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:16:26.460545 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:26.460504    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 31 16:16:26.461134 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:26.460621    4549 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 31 16:16:27.464528 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:27.464499    4549 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:16:27.465476 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:16:27.465456    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:16:37.011862 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:16:37.011822    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:17:07.011304 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:17:07.011268    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:17:29.767901 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:29.767861    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:17:30.565496 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:30.565458    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:17:30.565746 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:30.565697    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" podUID="578093aa-2d02-4c82-8dbe-00f91514907e" containerName="maas-api" containerID="cri-o://7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516" gracePeriod=30
Oct 31 16:17:31.805956 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.805934    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:17:31.920046 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.919969    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\") pod \"578093aa-2d02-4c82-8dbe-00f91514907e\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") "
Oct 31 16:17:31.920046 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.920007    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") pod \"578093aa-2d02-4c82-8dbe-00f91514907e\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") "
Oct 31 16:17:31.920239 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.920125    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\" (UniqueName: \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\") pod \"578093aa-2d02-4c82-8dbe-00f91514907e\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") "
Oct 31 16:17:31.920239 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.920180    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\") pod \"578093aa-2d02-4c82-8dbe-00f91514907e\" (UID: \"578093aa-2d02-4c82-8dbe-00f91514907e\") "
Oct 31 16:17:31.920610 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.920553    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca" pod "578093aa-2d02-4c82-8dbe-00f91514907e" (UID: "578093aa-2d02-4c82-8dbe-00f91514907e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:31.922284 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.922247    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs" pod "578093aa-2d02-4c82-8dbe-00f91514907e" (UID: "578093aa-2d02-4c82-8dbe-00f91514907e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:31.922401 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.922279    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7" pod "578093aa-2d02-4c82-8dbe-00f91514907e" (UID: "578093aa-2d02-4c82-8dbe-00f91514907e"). InnerVolumeSpecName "kube-api-access-zkbr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:31.922401 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:31.922315    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls" pod "578093aa-2d02-4c82-8dbe-00f91514907e" (UID: "578093aa-2d02-4c82-8dbe-00f91514907e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:17:32.021205 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.021170    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:32.021205 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.021198    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/578093aa-2d02-4c82-8dbe-00f91514907e-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:32.021205 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.021209    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkbr7\" (UniqueName: \"kubernetes.io/projected/578093aa-2d02-4c82-8dbe-00f91514907e-kube-api-access-zkbr7\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:32.021440 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.021220    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/578093aa-2d02-4c82-8dbe-00f91514907e-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:32.360600 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.360551    4549 generic.go:356] "Generic (PLEG): container finished" podID="578093aa-2d02-4c82-8dbe-00f91514907e" containerID="7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516" exitCode=0
Oct 31 16:17:32.360813 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.360616    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" event={"ID":"578093aa-2d02-4c82-8dbe-00f91514907e","Type":"ContainerDied","Data":"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"}
Oct 31 16:17:32.360813 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.360632    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"
Oct 31 16:17:32.360813 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.360682    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf" event={"ID":"578093aa-2d02-4c82-8dbe-00f91514907e","Type":"ContainerDied","Data":"c478ec89f0f651ee3e9dd3e15a66d74d595069873a1220174d5d6606d2cb9f78"}
Oct 31 16:17:32.360813 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.360703    4549 scope.go:122] "RemoveContainer" containerID="7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"
Oct 31 16:17:32.371853 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.371833    4549 scope.go:122] "RemoveContainer" containerID="7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"
Oct 31 16:17:32.372213 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:17:32.372190    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516\": container with ID starting with 7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516 not found: ID does not exist" containerID="7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"
Oct 31 16:17:32.372316 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.372222    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516"} err="failed to get container status \"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516\": rpc error: code = NotFound desc = could not find container \"7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516\": container with ID starting with 7ee2f35fe8b562f1bd1fcd3947ccd6ce2413c06df2ab77ee1b7de96b6fdb4516 not found: ID does not exist"
Oct 31 16:17:32.524411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.524379    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:17:32.528598 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:32.528557    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14a7e5-5796c7c574-kdxnf"]
Oct 31 16:17:33.418600 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:33.418539    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578093aa-2d02-4c82-8dbe-00f91514907e" path="/var/lib/kubelet/pods/578093aa-2d02-4c82-8dbe-00f91514907e/volumes"
Oct 31 16:17:34.768650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:34.768604    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing" containerID="cri-o://1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa" gracePeriod=25
Oct 31 16:17:37.011676 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:17:37.011638    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:17:43.827428 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:43.827389    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:17:53.827439 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:53.827396    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 16:17:55.028468 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.028396    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:17:55.138712 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.138679    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\" (UniqueName: \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\") pod \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") "
Oct 31 16:17:55.138909 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.138732    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\") pod \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\" (UID: \"6a39c72d-aaf7-4fed-99e7-0ec8a6467139\") "
Oct 31 16:17:55.139206 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.139181    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume" pod "6a39c72d-aaf7-4fed-99e7-0ec8a6467139" (UID: "6a39c72d-aaf7-4fed-99e7-0ec8a6467139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:17:55.141849 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.141820    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq" pod "6a39c72d-aaf7-4fed-99e7-0ec8a6467139" (UID: "6a39c72d-aaf7-4fed-99e7-0ec8a6467139"). InnerVolumeSpecName "kube-api-access-5sjfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:17:55.239655 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.239617    4549 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-plugins-config-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.239655 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.239648    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sjfq\" (UniqueName: \"kubernetes.io/projected/6a39c72d-aaf7-4fed-99e7-0ec8a6467139-kube-api-access-5sjfq\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:17:55.489136 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.489104    4549 generic.go:356] "Generic (PLEG): container finished" podID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerID="1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa" exitCode=0
Oct 31 16:17:55.489347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.489150    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"
Oct 31 16:17:55.489347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.489189    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" event={"ID":"6a39c72d-aaf7-4fed-99e7-0ec8a6467139","Type":"ContainerDied","Data":"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"}
Oct 31 16:17:55.489347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.489241    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f" event={"ID":"6a39c72d-aaf7-4fed-99e7-0ec8a6467139","Type":"ContainerDied","Data":"f84ba4a85f6192221c9a29a57048a4bf0a11988ca253d985dddb210f1a6d7df0"}
Oct 31 16:17:55.489347 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.489271    4549 scope.go:122] "RemoveContainer" containerID="1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"
Oct 31 16:17:55.498331 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.498314    4549 scope.go:122] "RemoveContainer" containerID="1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"
Oct 31 16:17:55.498594 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:17:55.498553    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa\": container with ID starting with 1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa not found: ID does not exist" containerID="1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"
Oct 31 16:17:55.498689 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.498597    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa"} err="failed to get container status \"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa\": rpc error: code = NotFound desc = could not find container \"1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa\": container with ID starting with 1ecbb60605c5bd333f1603fb46abac41372c24554d1a5f09058349e504facdaa not found: ID does not exist"
Oct 31 16:17:55.652542 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.652513    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:17:55.701706 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:55.701638    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-14a7e5-56b77gzd2f"]
Oct 31 16:17:57.418332 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:17:57.418298    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" path="/var/lib/kubelet/pods/6a39c72d-aaf7-4fed-99e7-0ec8a6467139/volumes"
Oct 31 16:18:07.011503 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:18:07.011460    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:18:07.292143 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:07.292112    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:18:07.292357 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:07.292328    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" containerID="cri-o://e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e" gracePeriod=30
Oct 31 16:18:12.538602 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.538565    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:18:12.585713 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.585628    4549 generic.go:356] "Generic (PLEG): container finished" podID="92663597-71f8-449e-a2cc-24c91c36324d" containerID="e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e" exitCode=0
Oct 31 16:18:12.585713 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.585688    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"
Oct 31 16:18:12.585909 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.585720    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerDied","Data":"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"}
Oct 31 16:18:12.585909 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.585781    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" event={"ID":"92663597-71f8-449e-a2cc-24c91c36324d","Type":"ContainerDied","Data":"ca25e8fd2c83c70df79746b22dc53524a445bf92c80e92954bd81ae624471ba8"}
Oct 31 16:18:12.585909 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.585804    4549 scope.go:122] "RemoveContainer" containerID="e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"
Oct 31 16:18:12.595357 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.595338    4549 scope.go:122] "RemoveContainer" containerID="b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"
Oct 31 16:18:12.604127 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.604110    4549 scope.go:122] "RemoveContainer" containerID="e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"
Oct 31 16:18:12.604383 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:18:12.604365    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e\": container with ID starting with e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e not found: ID does not exist" containerID="e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"
Oct 31 16:18:12.604431 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.604393    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e"} err="failed to get container status \"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e\": rpc error: code = NotFound desc = could not find container \"e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e\": container with ID starting with e2d150b06bf156d91473fabcb0c637851e63ffcfb6cf3b0f84c9b025d9e8502e not found: ID does not exist"
Oct 31 16:18:12.604431 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.604409    4549 scope.go:122] "RemoveContainer" containerID="b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"
Oct 31 16:18:12.604683 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:18:12.604664    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7\": container with ID starting with b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7 not found: ID does not exist" containerID="b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"
Oct 31 16:18:12.604730 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.604690    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7"} err="failed to get container status \"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7\": rpc error: code = NotFound desc = could not find container \"b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7\": container with ID starting with b763d72d839d0062398b1b1ce964a7c71abb24f5464a7b7198ccabe2a0e39ef7 not found: ID does not exist"
Oct 31 16:18:12.605895 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.605878    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.605945 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.605917    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.605989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.605962    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.605989 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.605983    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606098 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606015    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606098 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606036    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606098 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606069    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606264 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606128    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606264 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606179    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\") pod \"92663597-71f8-449e-a2cc-24c91c36324d\" (UID: \"92663597-71f8-449e-a2cc-24c91c36324d\") "
Oct 31 16:18:12.606373 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606290    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:18:12.606373 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606313    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:18:12.606731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606431    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:18:12.606731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606508    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:18:12.606731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606538    4549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.606731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606558    4549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-data\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.606731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.606699    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:18:12.608353 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.608323    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "kube-api-access-w4mp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:12.608472 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.608451    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:18:12.608652 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.608617    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 16:18:12.608743 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.608727    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token" pod "92663597-71f8-449e-a2cc-24c91c36324d" (UID: "92663597-71f8-449e-a2cc-24c91c36324d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:18:12.707960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.707924    4549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-istio-token\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.707960 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.707960    4549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92663597-71f8-449e-a2cc-24c91c36324d-istiod-ca-cert\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.708212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.707975    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mp6\" (UniqueName: \"kubernetes.io/projected/92663597-71f8-449e-a2cc-24c91c36324d-kube-api-access-w4mp6\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.708212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.707991    4549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-istio-envoy\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.708212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.708005    4549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-credential-socket\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.708212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.708017    4549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92663597-71f8-449e-a2cc-24c91c36324d-workload-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:12.708212 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:12.708030    4549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92663597-71f8-449e-a2cc-24c91c36324d-istio-podinfo\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:18:13.006211 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:13.006172    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:18:13.011703 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:13.011667    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb"]
Oct 31 16:18:13.418852 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:13.418820    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92663597-71f8-449e-a2cc-24c91c36324d" path="/var/lib/kubelet/pods/92663597-71f8-449e-a2cc-24c91c36324d/volumes"
Oct 31 16:18:13.465584 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:13.465538    4549 patch_prober.go:29] interesting pod/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:18:13.465756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:18:13.465628    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-14a7e5-openshift-default-5d684c7c65mbtrb" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:18:37.011445 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:18:37.011403    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:19:07.011953 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:19:07.011914    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:19:26.483833 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:19:26.483783    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:19:26.484355 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:19:26.483857    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:19:37.012015 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:19:37.011971    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:19:41.489808 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:19:41.489716    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:19:41.490411 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:19:41.490378    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:20:07.012270 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:20:07.012221    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:20:25.046468 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:25.046431    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"]
Oct 31 16:20:25.095613 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:25.095559    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ab86fe-mrc6v"]
Oct 31 16:20:25.418139 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:25.418058    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91195bcb-1b77-4bbb-86b1-09de0be1698a" path="/var/lib/kubelet/pods/91195bcb-1b77-4bbb-86b1-09de0be1698a/volumes"
Oct 31 16:20:37.010955 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:20:37.010915    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:20:44.044948 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:44.044905    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"]
Oct 31 16:20:44.093753 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:44.093718    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00b18ff-4kslg"]
Oct 31 16:20:45.417681 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:20:45.417645    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124e9b4a-bb54-46b8-8df2-750dd48cc655" path="/var/lib/kubelet/pods/124e9b4a-bb54-46b8-8df2-750dd48cc655/volumes"
Oct 31 16:21:03.357877 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:21:03.357851    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:21:03.519383 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:21:03.519356    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:21:07.011045 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:21:07.011001    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:21:37.012442 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:21:37.012398    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:22:04.280359 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:22:04.280327    4549 scope.go:122] "RemoveContainer" containerID="6fb86f99d41b96d627f70df678e81330a23c51bdb8efb965a6b82025edeee354"
Oct 31 16:22:04.289820 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:22:04.289801    4549 scope.go:122] "RemoveContainer" containerID="8f62e8220ea6f01898d4ba1a89a8bb72e5bc5ef43be460f26f795eca802dfc20"
Oct 31 16:22:07.010806 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:22:07.010766    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:22:37.011476 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:22:37.011439    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:23:07.070488 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:23:07.070436    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:23:19.820606 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:23:19.817663    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:23:19.820606 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:23:19.817727    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:23:37.011249 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:23:37.011204    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:24:07.011060 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:24:07.011025    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:24:19.818772 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:19.818673    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:19.819546 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:19.819511    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:20.117652 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:20.116688    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:20.117652 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:20.116755    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:37.010813 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:24:37.010773    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:24:49.818596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:49.817689    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:49.818596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:49.817775    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:24:50.114804 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:50.114695    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:24:50.115135 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:24:50.115104    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:25:07.079603 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:25:07.078941    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:25:37.012190 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:25:37.012150    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:26:03.397847 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:26:03.397815    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:26:03.568740 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:26:03.568712    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:26:07.011288 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:26:07.011253    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:26:37.010959 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:26:37.010923    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:27:07.011296 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:27:07.011254    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:27:37.022636 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:27:37.022402    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:28:07.017282 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:28:07.017236    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:28:17.755605 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.755531    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756112    4549 cpu_manager.go:414] "containerMap: removing container" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756132    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756170    4549 cpu_manager.go:414] "containerMap: removing container" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing"
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756178    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing"
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756197    4549 cpu_manager.go:414] "containerMap: removing container" podUID="578093aa-2d02-4c82-8dbe-00f91514907e" containerName="maas-api"
Oct 31 16:28:17.756228 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756204    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578093aa-2d02-4c82-8dbe-00f91514907e" containerName="maas-api"
Oct 31 16:28:17.756561 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756297    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:28:17.756561 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756309    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a39c72d-aaf7-4fed-99e7-0ec8a6467139" containerName="payload-pre-processing"
Oct 31 16:28:17.756561 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.756321    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="578093aa-2d02-4c82-8dbe-00f91514907e" containerName="maas-api"
Oct 31 16:28:17.759458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.759439    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.816258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.816220    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:28:17.858480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.858451    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.858645 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.858485    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.858645 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.858595    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.858645 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.858627    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqxx\" (UniqueName: \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.862313 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.862290    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-bcf50e-serving-cert\""
Oct 31 16:28:17.862429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.862374    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:28:17.862429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.862385    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:28:17.862429 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.862425    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 16:28:17.862616 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.862526    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:28:17.959052 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.959016    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.959258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.959062    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqxx\" (UniqueName: \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.959258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.959148    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.959258 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:28:17.959164    4549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-serving-cert: secret "maas-api-e2e-worker-single-main-bcf50e-serving-cert" not found
Oct 31 16:28:17.959258 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.959175    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.959258 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:28:17.959245    4549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls podName:86355338-7e52-4b6e-91b4-162500910d3c nodeName:}" failed. No retries permitted until 2026-10-31 16:28:18.459224876 +0000 UTC m=+6139.185505847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls") pod "maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" (UID: "86355338-7e52-4b6e-91b4-162500910d3c") : secret "maas-api-e2e-worker-single-main-bcf50e-serving-cert" not found
Oct 31 16:28:17.959943 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.959919    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:17.961663 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:17.961644    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:18.016392 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:18.016366    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqxx\" (UniqueName: \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:18.463259 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:18.463172    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:18.465804 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:18.465777    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:18.672340 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:18.672307    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:19.047984 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:19.047953    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:28:19.049316 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:28:19.049290    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86355338_7e52_4b6e_91b4_162500910d3c.slice/crio-4c8ac48ac8368bcfee98b708f1b05d1c6ae53db53b281ff109c4be9fc2ec1bcf WatchSource:0}: Error finding container 4c8ac48ac8368bcfee98b708f1b05d1c6ae53db53b281ff109c4be9fc2ec1bcf: Status 404 returned error can't find the container with id 4c8ac48ac8368bcfee98b708f1b05d1c6ae53db53b281ff109c4be9fc2ec1bcf
Oct 31 16:28:19.051067 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:19.051048    4549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 16:28:19.368422 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:19.368339    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" event={"ID":"86355338-7e52-4b6e-91b4-162500910d3c","Type":"ContainerStarted","Data":"4c8ac48ac8368bcfee98b708f1b05d1c6ae53db53b281ff109c4be9fc2ec1bcf"}
Oct 31 16:28:20.382914 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:20.382826    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" event={"ID":"86355338-7e52-4b6e-91b4-162500910d3c","Type":"ContainerStarted","Data":"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"}
Oct 31 16:28:20.383336 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:20.382925    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:20.497168 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:20.497116    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" podStartSLOduration=2.494108379 podStartE2EDuration="3.497102049s" totalImagesPullingTime="1.00299367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:28:20.49549915 +0000 UTC m=+6141.221780138" watchObservedRunningTime="2026-10-31 16:28:20.497102049 +0000 UTC m=+6141.223383027"
Oct 31 16:28:26.393541 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:28:26.393512    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:28:37.011325 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:28:37.011290    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:29:07.012993 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:29:07.012952    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:29:37.011016 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:29:37.010976    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:29:44.811476 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:44.811433    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:29:44.812221 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:44.811704    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" podUID="86355338-7e52-4b6e-91b4-162500910d3c" containerName="maas-api" containerID="cri-o://99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0" gracePeriod=30
Oct 31 16:29:46.057100 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.057061    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:29:46.165806 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.165726    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\" (UniqueName: \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\") pod \"86355338-7e52-4b6e-91b4-162500910d3c\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") "
Oct 31 16:29:46.165987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.165808    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") pod \"86355338-7e52-4b6e-91b4-162500910d3c\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") "
Oct 31 16:29:46.165987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.165869    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\") pod \"86355338-7e52-4b6e-91b4-162500910d3c\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") "
Oct 31 16:29:46.165987 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.165887    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\") pod \"86355338-7e52-4b6e-91b4-162500910d3c\" (UID: \"86355338-7e52-4b6e-91b4-162500910d3c\") "
Oct 31 16:29:46.166243 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.166213    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca" pod "86355338-7e52-4b6e-91b4-162500910d3c" (UID: "86355338-7e52-4b6e-91b4-162500910d3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:29:46.168020 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.167997    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx" pod "86355338-7e52-4b6e-91b4-162500910d3c" (UID: "86355338-7e52-4b6e-91b4-162500910d3c"). InnerVolumeSpecName "kube-api-access-2mqxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:29:46.168020 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.168005    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs" pod "86355338-7e52-4b6e-91b4-162500910d3c" (UID: "86355338-7e52-4b6e-91b4-162500910d3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:46.168151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.167995    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls" pod "86355338-7e52-4b6e-91b4-162500910d3c" (UID: "86355338-7e52-4b6e-91b4-162500910d3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:29:46.267480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.267444    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86355338-7e52-4b6e-91b4-162500910d3c-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:29:46.267480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.267474    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:29:46.267480 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.267484    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mqxx\" (UniqueName: \"kubernetes.io/projected/86355338-7e52-4b6e-91b4-162500910d3c-kube-api-access-2mqxx\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:29:46.267756 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.267493    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86355338-7e52-4b6e-91b4-162500910d3c-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:29:46.852014 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.851973    4549 generic.go:356] "Generic (PLEG): container finished" podID="86355338-7e52-4b6e-91b4-162500910d3c" containerID="99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0" exitCode=0
Oct 31 16:29:46.852207 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.852027    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"
Oct 31 16:29:46.852207 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.852054    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" event={"ID":"86355338-7e52-4b6e-91b4-162500910d3c","Type":"ContainerDied","Data":"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"}
Oct 31 16:29:46.852207 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.852115    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg" event={"ID":"86355338-7e52-4b6e-91b4-162500910d3c","Type":"ContainerDied","Data":"4c8ac48ac8368bcfee98b708f1b05d1c6ae53db53b281ff109c4be9fc2ec1bcf"}
Oct 31 16:29:46.852207 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.852135    4549 scope.go:122] "RemoveContainer" containerID="99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"
Oct 31 16:29:46.861990 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.861966    4549 scope.go:122] "RemoveContainer" containerID="99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"
Oct 31 16:29:46.862246 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:29:46.862227    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0\": container with ID starting with 99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0 not found: ID does not exist" containerID="99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"
Oct 31 16:29:46.862293 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:46.862256    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0"} err="failed to get container status \"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0\": rpc error: code = NotFound desc = could not find container \"99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0\": container with ID starting with 99211f3b868ea50dd46e0afea77d0e948abc313120b80f767314d4daa11a01b0 not found: ID does not exist"
Oct 31 16:29:47.015277 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:47.015238    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:29:47.064534 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:47.064497    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bcf50e-7c5d5964-qhbjg"]
Oct 31 16:29:47.419052 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:29:47.419016    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86355338-7e52-4b6e-91b4-162500910d3c" path="/var/lib/kubelet/pods/86355338-7e52-4b6e-91b4-162500910d3c/volumes"
Oct 31 16:30:07.011863 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:30:07.011822    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:30:37.011064 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:30:37.010984    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:30:48.498035 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.497995    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 16:30:48.498701 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.498280    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" podUID="b301180c-09ce-4d9b-abf3-45e80856e298" containerName="manager" containerID="cri-o://cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da" gracePeriod=10
Oct 31 16:30:48.750719 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.750662    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 16:30:48.846149 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.846117    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\") pod \"b301180c-09ce-4d9b-abf3-45e80856e298\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") "
Oct 31 16:30:48.846332 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.846239    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\" (UniqueName: \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\") pod \"b301180c-09ce-4d9b-abf3-45e80856e298\" (UID: \"b301180c-09ce-4d9b-abf3-45e80856e298\") "
Oct 31 16:30:48.846556 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.846538    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume" pod "b301180c-09ce-4d9b-abf3-45e80856e298" (UID: "b301180c-09ce-4d9b-abf3-45e80856e298"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 16:30:48.848547 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.848528    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt" pod "b301180c-09ce-4d9b-abf3-45e80856e298" (UID: "b301180c-09ce-4d9b-abf3-45e80856e298"). InnerVolumeSpecName "kube-api-access-fgtvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:30:48.947071 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.947030    4549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b301180c-09ce-4d9b-abf3-45e80856e298-extensions-socket-volume\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:30:48.947071 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:48.947069    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtvt\" (UniqueName: \"kubernetes.io/projected/b301180c-09ce-4d9b-abf3-45e80856e298-kube-api-access-fgtvt\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:30:49.200430 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.200395    4549 generic.go:356] "Generic (PLEG): container finished" podID="b301180c-09ce-4d9b-abf3-45e80856e298" containerID="cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da" exitCode=0
Oct 31 16:30:49.200636 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.200454    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"
Oct 31 16:30:49.200636 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.200454    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" event={"ID":"b301180c-09ce-4d9b-abf3-45e80856e298","Type":"ContainerDied","Data":"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"}
Oct 31 16:30:49.200636 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.200614    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz" event={"ID":"b301180c-09ce-4d9b-abf3-45e80856e298","Type":"ContainerDied","Data":"193de42fdccd7366a6341d70602d64f92ccea690059f0055bb7d46ac54eea56b"}
Oct 31 16:30:49.200636 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.200637    4549 scope.go:122] "RemoveContainer" containerID="cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"
Oct 31 16:30:49.215125 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.215104    4549 scope.go:122] "RemoveContainer" containerID="cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"
Oct 31 16:30:49.215405 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:30:49.215384    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da\": container with ID starting with cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da not found: ID does not exist" containerID="cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"
Oct 31 16:30:49.215461 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.215415    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da"} err="failed to get container status \"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da\": rpc error: code = NotFound desc = could not find container \"cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da\": container with ID starting with cfa54903de53a55aaf8bc22ca8a32d91c577d63bffbb5644df0d1612f98b24da not found: ID does not exist"
Oct 31 16:30:49.364908 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.364874    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 16:30:49.417858 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:49.417829    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b7wgz"]
Oct 31 16:30:51.417636 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:30:51.417606    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b301180c-09ce-4d9b-abf3-45e80856e298" path="/var/lib/kubelet/pods/b301180c-09ce-4d9b-abf3-45e80856e298/volumes"
Oct 31 16:31:03.440151 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:03.440124    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:31:03.607523 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:03.607495    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:31:07.011777 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:31:07.011735    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:31:37.011216 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:31:37.011174    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:31:52.159264 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159232    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"]
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159665    4549 cpu_manager.go:414] "containerMap: removing container" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159680    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159689    4549 cpu_manager.go:414] "containerMap: removing container" podUID="86355338-7e52-4b6e-91b4-162500910d3c" containerName="maas-api"
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159694    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86355338-7e52-4b6e-91b4-162500910d3c" containerName="maas-api"
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159717    4549 cpu_manager.go:414] "containerMap: removing container" podUID="b301180c-09ce-4d9b-abf3-45e80856e298" containerName="manager"
Oct 31 16:31:52.159752 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159722    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b301180c-09ce-4d9b-abf3-45e80856e298" containerName="manager"
Oct 31 16:31:52.159958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159786    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="86355338-7e52-4b6e-91b4-162500910d3c" containerName="maas-api"
Oct 31 16:31:52.159958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159796    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="92663597-71f8-449e-a2cc-24c91c36324d" containerName="istio-proxy"
Oct 31 16:31:52.159958 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.159805    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="b301180c-09ce-4d9b-abf3-45e80856e298" containerName="manager"
Oct 31 16:31:52.162782 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.162767    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.216728 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.216700    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"]
Oct 31 16:31:52.225600 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.225549    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnglt\" (UniqueName: \"kubernetes.io/projected/c4ed16d7-04f7-49dd-aa36-7152a16643b3-kube-api-access-dnglt\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.225771 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.225674    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ed16d7-04f7-49dd-aa36-7152a16643b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.262348 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.262314    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jjrh5\""
Oct 31 16:31:52.326913 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.326876    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnglt\" (UniqueName: \"kubernetes.io/projected/c4ed16d7-04f7-49dd-aa36-7152a16643b3-kube-api-access-dnglt\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.326913 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.326920    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ed16d7-04f7-49dd-aa36-7152a16643b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.327285 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.327267    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4ed16d7-04f7-49dd-aa36-7152a16643b3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.383432 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.383398    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnglt\" (UniqueName: \"kubernetes.io/projected/c4ed16d7-04f7-49dd-aa36-7152a16643b3-kube-api-access-dnglt\") pod \"kuadrant-operator-controller-manager-678d87766f-g276d\" (UID: \"c4ed16d7-04f7-49dd-aa36-7152a16643b3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.475786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.475701    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:52.950162 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:52.950135    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"]
Oct 31 16:31:52.951286 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:31:52.951262    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ed16d7_04f7_49dd_aa36_7152a16643b3.slice/crio-157357ad8fed2f87a2a2dfe62b141e0e93fb39a1bbe6fce6279efbf396432b2d WatchSource:0}: Error finding container 157357ad8fed2f87a2a2dfe62b141e0e93fb39a1bbe6fce6279efbf396432b2d: Status 404 returned error can't find the container with id 157357ad8fed2f87a2a2dfe62b141e0e93fb39a1bbe6fce6279efbf396432b2d
Oct 31 16:31:53.528725 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:53.528688    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d" event={"ID":"c4ed16d7-04f7-49dd-aa36-7152a16643b3","Type":"ContainerStarted","Data":"6715a051b0af0d811b388ca3777a60746e5d0e47baaf3035dbb068f736c31295"}
Oct 31 16:31:53.529141 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:53.528747    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d" event={"ID":"c4ed16d7-04f7-49dd-aa36-7152a16643b3","Type":"ContainerStarted","Data":"157357ad8fed2f87a2a2dfe62b141e0e93fb39a1bbe6fce6279efbf396432b2d"}
Oct 31 16:31:53.529141 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:53.528820    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:31:53.742886 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:31:53.742801    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d" podStartSLOduration=1.742782874 podStartE2EDuration="1.742782874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:31:53.692379667 +0000 UTC m=+6354.418660645" watchObservedRunningTime="2026-10-31 16:31:53.742782874 +0000 UTC m=+6354.469063875"
Oct 31 16:32:04.536448 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:04.536412    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g276d"
Oct 31 16:32:07.010526 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:32:07.010482    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:32:37.012875 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:32:37.012830    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:32:41.483926 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:41.483863    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:32:41.484487 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:41.483973    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:32:55.613325 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.613275    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:32:55.616805 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.616784    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.718326 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.718292    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 16:32:55.718768 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.718428    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 16:32:55.719089 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.719068    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\""
Oct 31 16:32:55.719662 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.719642    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 16:32:55.720659 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.720639    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 16:32:55.722855 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.722832    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:32:55.741373 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.741340    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.741512 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.741405    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.741512 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.741492    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5bs\" (UniqueName: \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.741643 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.741523    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.842400 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.842368    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.842650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.842418    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.842650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.842542    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5bs\" (UniqueName: \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.842650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.842619    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.843275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.843252    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.845059 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.845035    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.845142 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.845113    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.898431 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.898359    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5bs\" (UniqueName: \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\") pod \"maas-api-84fff96864-cwwqn\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:55.929843 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:55.929801    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:56.301758 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:56.301729    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:32:56.303137 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:32:56.303109    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be9d561_b53c_48ec_93e5_75ac1e0286d5.slice/crio-3df4465ac13ac5095af7711c45f99eb6ecbe3e84d8d232f3e3f71faf7dd4c2a4 WatchSource:0}: Error finding container 3df4465ac13ac5095af7711c45f99eb6ecbe3e84d8d232f3e3f71faf7dd4c2a4: Status 404 returned error can't find the container with id 3df4465ac13ac5095af7711c45f99eb6ecbe3e84d8d232f3e3f71faf7dd4c2a4
Oct 31 16:32:56.896631 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:56.896594    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" event={"ID":"3be9d561-b53c-48ec-93e5-75ac1e0286d5","Type":"ContainerStarted","Data":"3df4465ac13ac5095af7711c45f99eb6ecbe3e84d8d232f3e3f71faf7dd4c2a4"}
Oct 31 16:32:57.903009 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:57.902976    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" event={"ID":"3be9d561-b53c-48ec-93e5-75ac1e0286d5","Type":"ContainerStarted","Data":"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"}
Oct 31 16:32:57.903436 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:57.903044    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:32:58.019926 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:32:58.017098    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" podStartSLOduration=2.105841397 podStartE2EDuration="3.016913089s" totalImagesPullingTime="911.071692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:32:58.01546323 +0000 UTC m=+6418.741744210" watchObservedRunningTime="2026-10-31 16:32:58.016913089 +0000 UTC m=+6418.743194067"
Oct 31 16:33:03.916044 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:03.916015    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:33:05.497264 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.497228    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:33:05.497751 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.497486    4549 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" podUID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" containerName="maas-api" containerID="cri-o://f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af" gracePeriod=30
Oct 31 16:33:05.546833 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.546793    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-g5489"]
Oct 31 16:33:05.551055 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.551036    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.596382 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.596349    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-g5489"]
Oct 31 16:33:05.638027 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.637994    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba2ab9c-46d2-4327-8b14-e0723411d32b-openshift-service-ca\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.638219 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.638034    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-maas-api-tls\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.638219 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.638154    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsqqc\" (UniqueName: \"kubernetes.io/projected/0ba2ab9c-46d2-4327-8b14-e0723411d32b-kube-api-access-rsqqc\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.638324 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.638227    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-metrics-certs\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.739000 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.738974    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba2ab9c-46d2-4327-8b14-e0723411d32b-openshift-service-ca\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.739138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.739012    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-maas-api-tls\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.739138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.739050    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsqqc\" (UniqueName: \"kubernetes.io/projected/0ba2ab9c-46d2-4327-8b14-e0723411d32b-kube-api-access-rsqqc\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.739138 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.739104    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-metrics-certs\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.739834 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.739799    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba2ab9c-46d2-4327-8b14-e0723411d32b-openshift-service-ca\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.741672 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.741651    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-maas-api-tls\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.741768 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.741749    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba2ab9c-46d2-4327-8b14-e0723411d32b-metrics-certs\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.742871 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.742850    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:33:05.795286 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.795254    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsqqc\" (UniqueName: \"kubernetes.io/projected/0ba2ab9c-46d2-4327-8b14-e0723411d32b-kube-api-access-rsqqc\") pod \"maas-api-7468745db8-g5489\" (UID: \"0ba2ab9c-46d2-4327-8b14-e0723411d32b\") " pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.840414 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.840377    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\" (UniqueName: \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\") pod \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") "
Oct 31 16:33:05.840654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.840451    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\") pod \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") "
Oct 31 16:33:05.840654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.840492    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\") pod \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") "
Oct 31 16:33:05.840654 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.840528    4549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\") pod \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\" (UID: \"3be9d561-b53c-48ec-93e5-75ac1e0286d5\") "
Oct 31 16:33:05.840962 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.840930    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca" pod "3be9d561-b53c-48ec-93e5-75ac1e0286d5" (UID: "3be9d561-b53c-48ec-93e5-75ac1e0286d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 16:33:05.842811 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.842787    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls" pod "3be9d561-b53c-48ec-93e5-75ac1e0286d5" (UID: "3be9d561-b53c-48ec-93e5-75ac1e0286d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:33:05.842888 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.842807    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs" pod "3be9d561-b53c-48ec-93e5-75ac1e0286d5" (UID: "3be9d561-b53c-48ec-93e5-75ac1e0286d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 16:33:05.842888 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.842825    4549 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs" pod "3be9d561-b53c-48ec-93e5-75ac1e0286d5" (UID: "3be9d561-b53c-48ec-93e5-75ac1e0286d5"). InnerVolumeSpecName "kube-api-access-wg5bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 16:33:05.883982 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.883955    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:05.941652 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.941618    4549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5bs\" (UniqueName: \"kubernetes.io/projected/3be9d561-b53c-48ec-93e5-75ac1e0286d5-kube-api-access-wg5bs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:33:05.941786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.941661    4549 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-metrics-certs\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:33:05.941786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.941678    4549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3be9d561-b53c-48ec-93e5-75ac1e0286d5-maas-api-tls\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:33:05.941786 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.941692    4549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3be9d561-b53c-48ec-93e5-75ac1e0286d5-openshift-service-ca\") on node \"ip-10-0-139-115.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 16:33:05.952701 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.952666    4549 generic.go:356] "Generic (PLEG): container finished" podID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" containerID="f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af" exitCode=0
Oct 31 16:33:05.952864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.952710    4549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"
Oct 31 16:33:05.952864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.952746    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" event={"ID":"3be9d561-b53c-48ec-93e5-75ac1e0286d5","Type":"ContainerDied","Data":"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"}
Oct 31 16:33:05.952864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.952802    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-cwwqn" event={"ID":"3be9d561-b53c-48ec-93e5-75ac1e0286d5","Type":"ContainerDied","Data":"3df4465ac13ac5095af7711c45f99eb6ecbe3e84d8d232f3e3f71faf7dd4c2a4"}
Oct 31 16:33:05.952864 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.952825    4549 scope.go:122] "RemoveContainer" containerID="f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"
Oct 31 16:33:05.966339 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.966286    4549 scope.go:122] "RemoveContainer" containerID="f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"
Oct 31 16:33:05.967812 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:33:05.967719    4549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af\": container with ID starting with f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af not found: ID does not exist" containerID="f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"
Oct 31 16:33:05.967812 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:05.967757    4549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af"} err="failed to get container status \"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af\": rpc error: code = NotFound desc = could not find container \"f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af\": container with ID starting with f8b51e5637c074a110937c76c2ca727213b0b5ce2cb4aed8fb00f2afc77da5af not found: ID does not exist"
Oct 31 16:33:06.115682 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:06.115560    4549 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:33:06.118948 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:06.118921    4549 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-cwwqn"]
Oct 31 16:33:06.255251 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:06.255225    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7468745db8-g5489"]
Oct 31 16:33:06.255932 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:33:06.255908    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba2ab9c_46d2_4327_8b14_e0723411d32b.slice/crio-9f4dd1a24a854c8341d5c962e1ee204810be46408db323a011275b4e05cdd267 WatchSource:0}: Error finding container 9f4dd1a24a854c8341d5c962e1ee204810be46408db323a011275b4e05cdd267: Status 404 returned error can't find the container with id 9f4dd1a24a854c8341d5c962e1ee204810be46408db323a011275b4e05cdd267
Oct 31 16:33:06.960186 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:06.960108    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489" event={"ID":"0ba2ab9c-46d2-4327-8b14-e0723411d32b","Type":"ContainerStarted","Data":"9f4dd1a24a854c8341d5c962e1ee204810be46408db323a011275b4e05cdd267"}
Oct 31 16:33:07.013703 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:33:07.013667    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:33:07.419335 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:07.419299    4549 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" path="/var/lib/kubelet/pods/3be9d561-b53c-48ec-93e5-75ac1e0286d5/volumes"
Oct 31 16:33:07.968207 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:07.968168    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489" event={"ID":"0ba2ab9c-46d2-4327-8b14-e0723411d32b","Type":"ContainerStarted","Data":"631476e57e761477d532c89164bbf30e2faefbb466a68585184556835771760b"}
Oct 31 16:33:07.968716 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:07.968245    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:08.082002 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:08.081942    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489" podStartSLOduration=2.281118175 podStartE2EDuration="3.08192653s" totalImagesPullingTime="800.808355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:08.080465698 +0000 UTC m=+6428.806746680" watchObservedRunningTime="2026-10-31 16:33:08.08192653 +0000 UTC m=+6428.808207508"
Oct 31 16:33:11.487810 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:11.487721    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:33:11.488206 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:11.487795    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:33:13.983835 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:33:13.983806    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7468745db8-g5489"
Oct 31 16:33:37.012219 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:33:37.012180    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:34:07.011333 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:34:07.011288    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:34:37.010707 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:34:37.010668    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:35:07.013694 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:35:07.013648    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:35:37.013294 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:35:37.013233    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:36:03.483501 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:36:03.483475    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:36:03.682300 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:36:03.682277    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847akl7w7_f44a9d6e-bf5a-4a9d-91b4-9733ed519a82/extract/0.log"
Oct 31 16:36:07.012796 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:36:07.012752    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:36:37.072591 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:36:37.072461    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:37:07.010527 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:37:07.010493    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:37:34.827735 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:34.827690    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:37:34.828542 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:34.828506    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:37:35.117650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:35.117079    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:37:35.117650 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:35.117159    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:37:37.011359 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:37:37.011317    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:37:56.491798 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:56.491692    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-szjt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:37:56.492406 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:37:56.492370    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-szjt6" podUID="d2a18e51-b0aa-4cf8-ae11-961e6f757e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:38:07.081108 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:38:07.081067    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:38:37.011615 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:38:37.011548    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:38:49.817743 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:38:49.817693    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hrqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:38:49.818556 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:38:49.818518    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hrqdl" podUID="9bc6f8c7-eb08-4416-9cc7-18b828abcca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:38:50.114656 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:38:50.110739    4549 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-nzdpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 16:38:50.114656 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:38:50.110818    4549 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-nzdpc" podUID="4e7d9582-0fa5-452b-8160-297ef97ec4d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 16:39:07.011900 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:39:07.011854    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:39:28.297079 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:28.296984    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fc9km_fac77ca4-d950-4750-9ed3-3c8550b6549e/manager/0.log"
Oct 31 16:39:31.099871 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:31.099837    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7468745db8-g5489_0ba2ab9c-46d2-4327-8b14-e0723411d32b/maas-api/0.log"
Oct 31 16:39:37.011161 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:39:37.011126    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:39:37.612202 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:37.612152    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z7xph_fe94f527-0b08-478a-baa0-4f92659bf5d6/manager/0.log"
Oct 31 16:39:37.924795 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:37.924706    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/1.log"
Oct 31 16:39:38.464471 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:38.464440    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g276d_c4ed16d7-04f7-49dd-aa36-7152a16643b3/manager/0.log"
Oct 31 16:39:39.988500 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:39.988468    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/0.log"
Oct 31 16:39:40.578193 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:40.578138    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b_08359732-c9bd-4176-94b3-f869c31849a4/istio-proxy/0.log"
Oct 31 16:39:41.783962 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:41.783929    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hrqdl_9bc6f8c7-eb08-4416-9cc7-18b828abcca1/istio-proxy/0.log"
Oct 31 16:39:42.570596 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:42.570546    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nzdpc_4e7d9582-0fa5-452b-8160-297ef97ec4d2/istio-proxy/1.log"
Oct 31 16:39:43.393824 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:43.393777    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-szjt6_d2a18e51-b0aa-4cf8-ae11-961e6f757e1b/istio-proxy/0.log"
Oct 31 16:39:44.082816 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:44.082791    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8_0774f12c-928b-4705-b474-3a0afa129dcf/istio-proxy/0.log"
Oct 31 16:39:45.518131 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:39:45.518107    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nzdpc_4e7d9582-0fa5-452b-8160-297ef97ec4d2/istio-proxy/0.log"
Oct 31 16:40:07.011677 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:40:07.011639    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:40:09.308041 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:09.308007    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hmc4m_32080cb0-02b5-44e5-a7b2-3f33c5f97f48/global-pull-secret-syncer/0.log"
Oct 31 16:40:11.672458 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:11.672424    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ktm59_ebed5d09-55bf-4935-8ea9-55360332d683/konnectivity-agent/0.log"
Oct 31 16:40:12.679317 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:12.679272    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fc9km_fac77ca4-d950-4750-9ed3-3c8550b6549e/manager/0.log"
Oct 31 16:40:13.437771 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:13.437731    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-115.us-west-1.compute.internal_d57afef5546aa1919f861928dcb50ce6/haproxy/0.log"
Oct 31 16:40:19.349210 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.349171    4549 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"]
Oct 31 16:40:19.349807 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.349787    4549 cpu_manager.go:414] "containerMap: removing container" podUID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" containerName="maas-api"
Oct 31 16:40:19.349895 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.349810    4549 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" containerName="maas-api"
Oct 31 16:40:19.349954 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.349935    4549 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be9d561-b53c-48ec-93e5-75ac1e0286d5" containerName="maas-api"
Oct 31 16:40:19.353030 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.353010    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.411086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.411052    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"]
Oct 31 16:40:19.419983 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.419956    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-proc\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.420222 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.419992    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-lib-modules\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.420222 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.420022    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-sys\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.420222 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.420113    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-podres\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.420222 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.420164    4549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm94s\" (UniqueName: \"kubernetes.io/projected/46653429-6ba8-41cd-8589-46ec766ee747-kube-api-access-wm94s\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.456622 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.456534    4549 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b9r2f\"/\"default-dockercfg-ztmvf\""
Oct 31 16:40:19.456622 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.456541    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"kube-root-ca.crt\""
Oct 31 16:40:19.456871 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.456633    4549 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"openshift-service-ca.crt\""
Oct 31 16:40:19.520731 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520690    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-podres\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520740    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm94s\" (UniqueName: \"kubernetes.io/projected/46653429-6ba8-41cd-8589-46ec766ee747-kube-api-access-wm94s\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520799    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-podres\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520837    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-proc\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520864    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-lib-modules\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520882    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-proc\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520899    4549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-sys\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520939    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-sys\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.521003 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.520974    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46653429-6ba8-41cd-8589-46ec766ee747-lib-modules\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.580701 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.580637    4549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm94s\" (UniqueName: \"kubernetes.io/projected/46653429-6ba8-41cd-8589-46ec766ee747-kube-api-access-wm94s\") pod \"perf-node-gather-daemonset-ncfj5\" (UID: \"46653429-6ba8-41cd-8589-46ec766ee747\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.647979 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.647858    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fc9km_fac77ca4-d950-4750-9ed3-3c8550b6549e/manager/0.log"
Oct 31 16:40:19.665978 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.665949    4549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:19.788730 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:19.788702    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z7xph_fe94f527-0b08-478a-baa0-4f92659bf5d6/manager/0.log"
Oct 31 16:40:20.038066 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.038043    4549 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"]
Oct 31 16:40:20.039332 ip-10-0-139-115 kubenswrapper[4549]: W1031 16:40:20.039306    4549 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46653429_6ba8_41cd_8589_46ec766ee747.slice/crio-f74ee75ca192d0d95352c3ffb5b9b93035e8b3f10cf1a7091efa0c449c00a50a WatchSource:0}: Error finding container f74ee75ca192d0d95352c3ffb5b9b93035e8b3f10cf1a7091efa0c449c00a50a: Status 404 returned error can't find the container with id f74ee75ca192d0d95352c3ffb5b9b93035e8b3f10cf1a7091efa0c449c00a50a
Oct 31 16:40:20.207950 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.207920    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/1.log"
Oct 31 16:40:20.208137 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.207999    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/0.log"
Oct 31 16:40:20.929830 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.929792    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5" event={"ID":"46653429-6ba8-41cd-8589-46ec766ee747","Type":"ContainerStarted","Data":"81ddd12503160cd7f5e5252876a7a8a5a276f71baa8a97d906aa739605d14075"}
Oct 31 16:40:20.930268 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.929857    4549 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5" event={"ID":"46653429-6ba8-41cd-8589-46ec766ee747","Type":"ContainerStarted","Data":"f74ee75ca192d0d95352c3ffb5b9b93035e8b3f10cf1a7091efa0c449c00a50a"}
Oct 31 16:40:20.930268 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.929995    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:20.981749 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:20.981718    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g276d_c4ed16d7-04f7-49dd-aa36-7152a16643b3/manager/0.log"
Oct 31 16:40:21.047338 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:21.047270    4549 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5" podStartSLOduration=2.047251465 podStartE2EDuration="2.047251465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:21.044440444 +0000 UTC m=+6861.770721421" watchObservedRunningTime="2026-10-31 16:40:21.047251465 +0000 UTC m=+6861.773532457"
Oct 31 16:40:24.420915 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:24.420885    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/alertmanager/0.log"
Oct 31 16:40:24.837585 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:24.837546    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/config-reloader/0.log"
Oct 31 16:40:25.255086 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:25.254956    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/kube-rbac-proxy-web/0.log"
Oct 31 16:40:25.864764 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:25.864732    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/kube-rbac-proxy/0.log"
Oct 31 16:40:25.955235 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:25.955199    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-fc9km_fac77ca4-d950-4750-9ed3-3c8550b6549e/manager/0.log"
Oct 31 16:40:26.281555 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:26.281526    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/kube-rbac-proxy-metric/0.log"
Oct 31 16:40:26.699333 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:26.699252    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/prom-label-proxy/0.log"
Oct 31 16:40:26.948030 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:26.948002    4549 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-ncfj5"
Oct 31 16:40:27.117028 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:27.116990    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c848632e-78b5-405a-ba1a-b7b22fac7a91/init-config-reloader/0.log"
Oct 31 16:40:27.972848 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:27.972817    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqxph_d8c6841a-9e72-4b90-a387-012c684b5e2f/kube-state-metrics/0.log"
Oct 31 16:40:28.385419 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:28.385373    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqxph_d8c6841a-9e72-4b90-a387-012c684b5e2f/kube-rbac-proxy-main/0.log"
Oct 31 16:40:28.801766 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:28.801729    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jqxph_d8c6841a-9e72-4b90-a387-012c684b5e2f/kube-rbac-proxy-self/0.log"
Oct 31 16:40:29.221447 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:29.221371    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5df946bc5-wfb69_289504d6-1764-433c-b803-3ce22a49f070/metrics-server/0.log"
Oct 31 16:40:31.483970 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:31.483943    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6f89d9f6ccjck8b_08359732-c9bd-4176-94b3-f869c31849a4/istio-proxy/0.log"
Oct 31 16:40:32.553649 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:32.553619    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hrqdl_9bc6f8c7-eb08-4416-9cc7-18b828abcca1/istio-proxy/0.log"
Oct 31 16:40:32.571939 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:32.571909    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmtt5_202b35c4-ef2a-42e3-b975-233bdcb69bdb/node-exporter/0.log"
Oct 31 16:40:32.986785 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:32.986713    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmtt5_202b35c4-ef2a-42e3-b975-233bdcb69bdb/kube-rbac-proxy/0.log"
Oct 31 16:40:33.199234 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:33.199190    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nzdpc_4e7d9582-0fa5-452b-8160-297ef97ec4d2/istio-proxy/0.log"
Oct 31 16:40:33.200690 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:33.200667    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-nzdpc_4e7d9582-0fa5-452b-8160-297ef97ec4d2/istio-proxy/1.log"
Oct 31 16:40:33.402188 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:33.402160    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xmtt5_202b35c4-ef2a-42e3-b975-233bdcb69bdb/init-textfile/0.log"
Oct 31 16:40:33.737723 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:33.737646    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-szjt6_d2a18e51-b0aa-4cf8-ae11-961e6f757e1b/istio-proxy/0.log"
Oct 31 16:40:34.397974 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:34.397945    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-fndl8_0774f12c-928b-4705-b474-3a0afa129dcf/istio-proxy/0.log"
Oct 31 16:40:37.011434 ip-10-0-139-115 kubenswrapper[4549]: E1031 16:40:37.011391    4549 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jk8rn" containerName="extractor"
Oct 31 16:40:38.043092 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:38.043066    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xncbp_ba08ded1-719d-4f0c-826a-d5b47b06eded/prometheus-operator/0.log"
Oct 31 16:40:38.497179 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:38.497108    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-xncbp_ba08ded1-719d-4f0c-826a-d5b47b06eded/kube-rbac-proxy/0.log"
Oct 31 16:40:38.914983 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:38.914958    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-9kpgn_663b9426-03ca-4551-b31a-3a5039707fa9/prometheus-operator-admission-webhook/0.log"
Oct 31 16:40:40.970032 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:40.969954    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-z7xph_fe94f527-0b08-478a-baa0-4f92659bf5d6/manager/0.log"
Oct 31 16:40:41.182926 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:41.182894    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/1.log"
Oct 31 16:40:41.182926 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:41.182916    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-52jvs_fa6cac91-ffce-4ec7-8c7d-1fc35213fef7/manager/0.log"
Oct 31 16:40:41.524275 ip-10-0-139-115 kubenswrapper[4549]: I1031 16:40:41.524245    4549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g276d_c4ed16d7-04f7-49dd-aa36-7152a16643b3/manager/0.log"